[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:09:55 dist-test-slave-n3fx krb5kdc[9844](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:09:55 dist-test-slave-n3fx krb5kdc[9844](info): set up 2 sockets
Jun 20 19:09:55 dist-test-slave-n3fx krb5kdc[9844](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260620 19:09:56.220530 9837 mini_kdc.cc:121] Time spent starting KDC: real 0.952s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/krb5kdc/test-user.keytab.
Jun 20 19:09:56 dist-test-slave-n3fx krb5kdc[9844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:09:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:09:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.155.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.155.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.155.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab.
Entry for principal kudu/127.9.155.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab.
I20260620 19:09:56.343437 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:38609
--webserver_interface=127.9.155.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab
--principal=kudu/127.9.155.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:33911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.155.126:38609
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:09:56.453712 9857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:09:56.453984 9857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:09:56.454023 9857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:09:56.458005 9857 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:09:56.458066 9857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:09:56.458086 9857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:09:56.458101 9857 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:09:56.458115 9857 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:09:56.463405 9857 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:33911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.155.126:38609
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab
--principal=kudu/127.9.155.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:38609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.155.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9857
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:09:56.464509 9857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:09:56.465510 9857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:09:56.474025 9865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:56.474077 9862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:56.474104 9863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:09:56.474175 9857 server_base.cc:1034] running on GCE node
I20260620 19:09:56.474825 9857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:09:56.475898 9857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:09:56.477051 9857 hybrid_clock.cc:648] HybridClock initialized: now 1781982596477026 us; error 39 us; skew 500 ppm
Jun 20 19:09:56 dist-test-slave-n3fx krb5kdc[9844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982596, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:09:56.479689 9857 init.cc:377] Logged in from keytab as kudu/127.9.155.126@KRBTEST.COM (short username kudu)
I20260620 19:09:56.480640 9857 webserver.cc:460] Webserver started at http://127.9.155.126:39173/ using document root <none> and password file <none>
I20260620 19:09:56.481175 9857 fs_manager.cc:359] Metadata directory not provided
I20260620 19:09:56.481238 9857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:09:56.481423 9857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:09:56.483397 9857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/minicluster-data/master-0/data/instance:
uuid: "bbf0b13eee894ef5acbcd13d42f3b0e3"
format_stamp: "Formatted at 2026-06-20 19:09:56 on dist-test-slave-n3fx"
server_key: "0942b2aaeab0e863c9cf3961112548ca"
server_key_iv: "217235daf7353650fe8b551822c00535"
server_key_version: "encryptionkey@0"
I20260620 19:09:56.483939 9857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/minicluster-data/master-0/wal/instance:
uuid: "bbf0b13eee894ef5acbcd13d42f3b0e3"
format_stamp: "Formatted at 2026-06-20 19:09:56 on dist-test-slave-n3fx"
server_key: "0942b2aaeab0e863c9cf3961112548ca"
server_key_iv: "217235daf7353650fe8b551822c00535"
server_key_version: "encryptionkey@0"
I20260620 19:09:56.487541 9857 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 19:09:56.489996 9872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:56.491309 9857 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 19:09:56.491472 9857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/minicluster-data/master-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/minicluster-data/master-0/wal
uuid: "bbf0b13eee894ef5acbcd13d42f3b0e3"
format_stamp: "Formatted at 2026-06-20 19:09:56 on dist-test-slave-n3fx"
server_key: "0942b2aaeab0e863c9cf3961112548ca"
server_key_iv: "217235daf7353650fe8b551822c00535"
server_key_version: "encryptionkey@0"
I20260620 19:09:56.491581 9857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:09:56.500756 9857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:09:56.503672 9857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:09:56.503844 9857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:09:56.510521 9857 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.126:38609
I20260620 19:09:56.510555 9924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.126:38609 every 8 connection(s)
I20260620 19:09:56.511518 9857 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
I20260620 19:09:56.514575 9925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:09:56.519975 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 9857
I20260620 19:09:56.520167 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/minicluster-data/master-0/wal/instance
I20260620 19:09:56.521294 9925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbf0b13eee894ef5acbcd13d42f3b0e3: Bootstrap starting.
I20260620 19:09:56.521893 9837 external_mini_cluster.cc:1428] Setting key 23689880c09ac249e3e5134b3b0f62e0
I20260620 19:09:56.524598 9925 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bbf0b13eee894ef5acbcd13d42f3b0e3: Neither blocks nor log segments found. Creating new log.
I20260620 19:09:56.525475 9925 log.cc:824] T 00000000000000000000000000000000 P bbf0b13eee894ef5acbcd13d42f3b0e3: Log is configured to *not* fsync() on all Append() calls
I20260620 19:09:56.527614 9925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbf0b13eee894ef5acbcd13d42f3b0e3: No bootstrap required, opened a new log
I20260620 19:09:56.531805 9925 raft_consensus.cc:348] T 00000000000000000000000000000000 P bbf0b13eee894ef5acbcd13d42f3b0e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbf0b13eee894ef5acbcd13d42f3b0e3" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 38609 } }
I20260620 19:09:56.532116 9925 raft_consensus.cc:374] T 00000000000000000000000000000000 P bbf0b13eee894ef5acbcd13d42f3b0e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:09:56.532179 9925 raft_consensus.cc:729] T 00000000000000000000000000000000 P bbf0b13eee894ef5acbcd13d42f3b0e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf0b13eee894ef5acbcd13d42f3b0e3, State: Initialized, Role: FOLLOWER
I20260620 19:09:56.532697 9925 consensus_queue.cc:260] T 00000000000000000000000000000000 P bbf0b13eee894ef5acbcd13d42f3b0e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbf0b13eee894ef5acbcd13d42f3b0e3" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 38609 } }
I20260620 19:09:56.532835 9925 raft_consensus.cc:388] T 00000000000000000000000000000000 P bbf0b13eee894ef5acbcd13d42f3b0e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:09:56.532896 9925 raft_consensus.cc:482] T 00000000000000000000000000000000 P bbf0b13eee894ef5acbcd13d42f3b0e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:09:56.532994 9925 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bbf0b13eee894ef5acbcd13d42f3b0e3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:09:56.533946 9925 raft_consensus.cc:504] T 00000000000000000000000000000000 P bbf0b13eee894ef5acbcd13d42f3b0e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbf0b13eee894ef5acbcd13d42f3b0e3" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 38609 } }
I20260620 19:09:56.534291 9925 leader_election.cc:302] T 00000000000000000000000000000000 P bbf0b13eee894ef5acbcd13d42f3b0e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bbf0b13eee894ef5acbcd13d42f3b0e3; no voters:
I20260620 19:09:56.534593 9925 leader_election.cc:288] T 00000000000000000000000000000000 P bbf0b13eee894ef5acbcd13d42f3b0e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:09:56.534718 9930 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bbf0b13eee894ef5acbcd13d42f3b0e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:09:56.534950 9930 raft_consensus.cc:686] T 00000000000000000000000000000000 P bbf0b13eee894ef5acbcd13d42f3b0e3 [term 1 LEADER]: Becoming Leader. State: Replica: bbf0b13eee894ef5acbcd13d42f3b0e3, State: Running, Role: LEADER
I20260620 19:09:56.535300 9930 consensus_queue.cc:237] T 00000000000000000000000000000000 P bbf0b13eee894ef5acbcd13d42f3b0e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbf0b13eee894ef5acbcd13d42f3b0e3" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 38609 } }
I20260620 19:09:56.535820 9925 sys_catalog.cc:564] T 00000000000000000000000000000000 P bbf0b13eee894ef5acbcd13d42f3b0e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:09:56.537034 9931 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbf0b13eee894ef5acbcd13d42f3b0e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bbf0b13eee894ef5acbcd13d42f3b0e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbf0b13eee894ef5acbcd13d42f3b0e3" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 38609 } } }
I20260620 19:09:56.537168 9931 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbf0b13eee894ef5acbcd13d42f3b0e3 [sys.catalog]: This master's current role is: LEADER
I20260620 19:09:56.537406 9931 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbf0b13eee894ef5acbcd13d42f3b0e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bbf0b13eee894ef5acbcd13d42f3b0e3. Latest consensus state: current_term: 1 leader_uuid: "bbf0b13eee894ef5acbcd13d42f3b0e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbf0b13eee894ef5acbcd13d42f3b0e3" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 38609 } } }
I20260620 19:09:56.537493 9931 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbf0b13eee894ef5acbcd13d42f3b0e3 [sys.catalog]: This master's current role is: LEADER
I20260620 19:09:56.537779 9939 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:09:56.541342 9939 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 19:09:56 dist-test-slave-n3fx krb5kdc[9844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:09:56.548269 9939 catalog_manager.cc:1347] Generated new cluster ID: 13d71422a972488db2bb493d4351d449
I20260620 19:09:56.548367 9939 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:09:56.549094 9928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:56.525567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50158 (local address 127.9.155.126:38609)
0620 19:09:56.526097 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:56.526118 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:56.527418 (+ 1300us) server_negotiation.cc:407] Connection header received
0620 19:09:56.531294 (+ 3876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:56.531321 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:56.531774 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:56.532171 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:56.534849 (+ 2678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:56.536394 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:56.538233 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:56.538939 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:56.539097 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:56.542932 (+ 3835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:56.542938 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:56.542951 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:09:56.542983 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:56.544648 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:56.545188 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:56.545193 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:56.545197 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:56.545328 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:56.545687 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:56.545690 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:56.545692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:56.546241 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:56.546386 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:56.547271 (+ 885us) server_negotiation.cc:299] Negotiation successful
0620 19:09:56.547609 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":180,"threads_started":1}
I20260620 19:09:56.559036 9939 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:09:56.559930 9939 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:09:56.564584 9939 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bbf0b13eee894ef5acbcd13d42f3b0e3: Generated new TSK 0
I20260620 19:09:56.565196 9939 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.155.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.155.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.155.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.65.keytab.
Entry for principal kudu/127.9.155.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.65.keytab.
I20260620 19:09:56.593046 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.65:0
--local_ip_for_outbound_sockets=127.9.155.65
--webserver_interface=127.9.155.65
--webserver_port=0
--tserver_master_addrs=127.9.155.126:38609
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.65.keytab
--principal=kudu/127.9.155.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:33911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:09:56.713686 9951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:09:56.713984 9951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:09:56.714030 9951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:09:56.718221 9951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:09:56.718295 9951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:09:56.718360 9951 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.65
I20260620 19:09:56.723835 9951 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:33911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.65.keytab
--principal=kudu/127.9.155.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.155.65
--webserver_port=0
--tserver_master_addrs=127.9.155.126:38609
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9951
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.65
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:09:56.724886 9951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:09:56.725836 9951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:09:56.735370 9959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:56.735378 9956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:56.735428 9957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:09:56.735560 9951 server_base.cc:1034] running on GCE node
I20260620 19:09:56.736112 9951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:09:56.736802 9951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:09:56.737985 9951 hybrid_clock.cc:648] HybridClock initialized: now 1781982596737955 us; error 41 us; skew 500 ppm
Jun 20 19:09:56 dist-test-slave-n3fx krb5kdc[9844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982596, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:09:56.740707 9951 init.cc:377] Logged in from keytab as kudu/127.9.155.65@KRBTEST.COM (short username kudu)
I20260620 19:09:56.741577 9951 webserver.cc:460] Webserver started at http://127.9.155.65:41947/ using document root <none> and password file <none>
I20260620 19:09:56.742097 9951 fs_manager.cc:359] Metadata directory not provided
I20260620 19:09:56.742161 9951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:09:56.742355 9951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:09:56.744359 9951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/minicluster-data/ts-0/data/instance:
uuid: "baa678244d1346bdb5b5c97d44309344"
format_stamp: "Formatted at 2026-06-20 19:09:56 on dist-test-slave-n3fx"
server_key: "7c8f06029a1c4b463c4a4f9aff9baae8"
server_key_iv: "70bbd9820c907fd3e20e782e449e0fcd"
server_key_version: "encryptionkey@0"
I20260620 19:09:56.744913 9951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/minicluster-data/ts-0/wal/instance:
uuid: "baa678244d1346bdb5b5c97d44309344"
format_stamp: "Formatted at 2026-06-20 19:09:56 on dist-test-slave-n3fx"
server_key: "7c8f06029a1c4b463c4a4f9aff9baae8"
server_key_iv: "70bbd9820c907fd3e20e782e449e0fcd"
server_key_version: "encryptionkey@0"
I20260620 19:09:56.748638 9951 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:09:56.751032 9966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:56.752400 9951 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:09:56.752512 9951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/minicluster-data/ts-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/minicluster-data/ts-0/wal
uuid: "baa678244d1346bdb5b5c97d44309344"
format_stamp: "Formatted at 2026-06-20 19:09:56 on dist-test-slave-n3fx"
server_key: "7c8f06029a1c4b463c4a4f9aff9baae8"
server_key_iv: "70bbd9820c907fd3e20e782e449e0fcd"
server_key_version: "encryptionkey@0"
I20260620 19:09:56.752616 9951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:09:56.768500 9951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:09:56.771375 9951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:09:56.771554 9951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:09:56.772137 9951 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:09:56.772964 9951 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:09:56.773012 9951 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:56.773064 9951 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:09:56.773093 9951 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:56.781309 9951 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.65:34949
I20260620 19:09:56.781366 10079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.65:34949 every 8 connection(s)
I20260620 19:09:56.782196 9951 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
I20260620 19:09:56.789830 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 9951
I20260620 19:09:56.789944 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/minicluster-data/ts-0/wal/instance
I20260620 19:09:56.790182 9837 external_mini_cluster.cc:1428] Setting key 56a52c28b036616c166065b0d5b180c2
Jun 20 19:09:56 dist-test-slave-n3fx krb5kdc[9844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982596, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.65@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:09:56.795857 9928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:56.784046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.65:40689 (local address 127.9.155.126:38609)
0620 19:09:56.784187 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:56.784191 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:56.785222 (+ 1031us) server_negotiation.cc:407] Connection header received
0620 19:09:56.786086 (+ 864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:56.786089 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:56.786146 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:56.786234 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:56.787369 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:56.787918 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:56.789066 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:56.789312 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:56.789401 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:56.792072 (+ 2671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:56.792075 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:56.792077 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:09:56.792100 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:56.793638 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:56.794137 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:56.794140 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:56.794142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:56.794227 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:56.794628 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:56.794631 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:56.794633 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:56.794850 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:56.794942 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:56.795549 (+ 607us) server_negotiation.cc:299] Negotiation successful
0620 19:09:56.795689 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 19:09:56.796697 10082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:56.784415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:38609 (local address 127.9.155.65:40689)
0620 19:09:56.785069 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:56.785109 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 19:09:56.785889 (+ 780us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:09:56.786412 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:09:56.786418 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:09:56.786771 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:09:56.787224 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:56.787238 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:56.788035 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:56.788038 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:56.788948 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:56.788959 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:56.789453 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:56.789457 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:56.789636 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:56.790246 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:09:56.790267 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:09:56.791910 (+ 1643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:09:56.793778 (+ 1868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:09:56.793783 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:09:56.793796 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:09:56.794043 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:09:56.794349 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:09:56.794352 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:09:56.794354 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:09:56.794524 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:09:56.794962 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:09:56.794968 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:09:56.795232 (+ 264us) client_negotiation.cc:769] Sending connection context
0620 19:09:56.795447 (+ 215us) client_negotiation.cc:240] Negotiation successful
0620 19:09:56.795783 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":117,"threads_started":1}
I20260620 19:09:56.798192 10080 heartbeater.cc:347] Connected to a master server at 127.9.155.126:38609
I20260620 19:09:56.798429 10080 heartbeater.cc:464] Registering TS with master...
I20260620 19:09:56.798911 10080 heartbeater.cc:511] Master 127.9.155.126:38609 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.9.155.66@KRBTEST.COM; defaulting to no policy
I20260620 19:09:56.800508 9889 ts_manager.cc:194] Registered new tserver with Master: baa678244d1346bdb5b5c97d44309344 (127.9.155.65:34949)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.155.66@KRBTEST.COM" created.
I20260620 19:09:56.801952 9889 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.155.65@KRBTEST.COM'} at 127.9.155.65:40689
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.155.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.66.keytab.
Entry for principal kudu/127.9.155.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.66.keytab.
I20260620 19:09:56.815701 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.66:0
--local_ip_for_outbound_sockets=127.9.155.66
--webserver_interface=127.9.155.66
--webserver_port=0
--tserver_master_addrs=127.9.155.126:38609
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.66.keytab
--principal=kudu/127.9.155.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:33911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:09:56.926878 10085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:09:56.927147 10085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:09:56.927210 10085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:09:56.931272 10085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:09:56.931336 10085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:09:56.931421 10085 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.66
I20260620 19:09:56.936684 10085 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:33911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.66.keytab
--principal=kudu/127.9.155.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.155.66
--webserver_port=0
--tserver_master_addrs=127.9.155.126:38609
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10085
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.66
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:09:56.937726 10085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:09:56.938759 10085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:09:56.948123 10090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:56.948128 10091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:56.948189 10093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:09:56.948382 10085 server_base.cc:1034] running on GCE node
I20260620 19:09:56.948782 10085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:09:56.949429 10085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:09:56.950582 10085 hybrid_clock.cc:648] HybridClock initialized: now 1781982596950552 us; error 43 us; skew 500 ppm
Jun 20 19:09:56 dist-test-slave-n3fx krb5kdc[9844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982596, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:09:56.953382 10085 init.cc:377] Logged in from keytab as kudu/127.9.155.66@KRBTEST.COM (short username kudu)
I20260620 19:09:56.954304 10085 webserver.cc:460] Webserver started at http://127.9.155.66:46687/ using document root <none> and password file <none>
I20260620 19:09:56.954852 10085 fs_manager.cc:359] Metadata directory not provided
I20260620 19:09:56.954921 10085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:09:56.955132 10085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:09:56.957237 10085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/minicluster-data/ts-1/data/instance:
uuid: "9eee0c5e3b524a6d8dd9474778b5011a"
format_stamp: "Formatted at 2026-06-20 19:09:56 on dist-test-slave-n3fx"
server_key: "4b1014c78b0e26c1961f0d94eda1821c"
server_key_iv: "a5ec48e5cf8e72236be00b89ae53fe3a"
server_key_version: "encryptionkey@0"
I20260620 19:09:56.957806 10085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/minicluster-data/ts-1/wal/instance:
uuid: "9eee0c5e3b524a6d8dd9474778b5011a"
format_stamp: "Formatted at 2026-06-20 19:09:56 on dist-test-slave-n3fx"
server_key: "4b1014c78b0e26c1961f0d94eda1821c"
server_key_iv: "a5ec48e5cf8e72236be00b89ae53fe3a"
server_key_version: "encryptionkey@0"
I20260620 19:09:56.961402 10085 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 19:09:56.963828 10100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:56.965086 10085 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:09:56.965190 10085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/minicluster-data/ts-1/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/minicluster-data/ts-1/wal
uuid: "9eee0c5e3b524a6d8dd9474778b5011a"
format_stamp: "Formatted at 2026-06-20 19:09:56 on dist-test-slave-n3fx"
server_key: "4b1014c78b0e26c1961f0d94eda1821c"
server_key_iv: "a5ec48e5cf8e72236be00b89ae53fe3a"
server_key_version: "encryptionkey@0"
I20260620 19:09:56.965343 10085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:09:56.982020 10085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:09:56.985139 10085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:09:56.985329 10085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:09:56.985916 10085 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:09:56.986740 10085 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:09:56.986786 10085 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:56.986840 10085 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:09:56.986869 10085 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:56.995702 10085 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.66:37319
I20260620 19:09:56.995761 10213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.66:37319 every 8 connection(s)
I20260620 19:09:56.996647 10085 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
I20260620 19:09:57.001899 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 10085
I20260620 19:09:57.002024 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/minicluster-data/ts-1/wal/instance
I20260620 19:09:57.002279 9837 external_mini_cluster.cc:1428] Setting key 613a3eeda1240cebbc3527bec78ba836
Jun 20 19:09:57 dist-test-slave-n3fx krb5kdc[9844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982596, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.66@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:09:57.010994 9928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:56.998471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.66:60375 (local address 127.9.155.126:38609)
0620 19:09:56.998626 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:56.998630 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:56.999677 (+ 1047us) server_negotiation.cc:407] Connection header received
0620 19:09:57.000638 (+ 961us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:57.000642 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:57.000724 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:57.000806 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:57.002061 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:57.002661 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:57.004053 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:57.004386 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:57.004487 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:57.007291 (+ 2804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:57.007294 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:57.007296 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:09:57.007326 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:57.008846 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:57.009446 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:57.009449 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:57.009450 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:57.009529 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:57.009885 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:57.009887 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:57.009889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:57.010050 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:57.010119 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:57.010746 (+ 627us) server_negotiation.cc:299] Negotiation successful
0620 19:09:57.010871 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
WARNING: no policy specified for kudu/127.9.155.67@KRBTEST.COM; defaulting to no policy
I20260620 19:09:57.012121 10216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:56.998891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:38609 (local address 127.9.155.66:60375)
0620 19:09:56.999526 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:56.999566 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 19:09:57.000441 (+ 875us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:09:57.000977 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:09:57.000984 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:09:57.001344 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:09:57.001875 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:57.001895 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:57.002786 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:57.002788 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:57.003936 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:57.003945 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:57.004533 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:57.004536 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:57.004680 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:57.005259 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:09:57.005281 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:09:57.007049 (+ 1768us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:09:57.008963 (+ 1914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:09:57.008969 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:09:57.008985 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:09:57.009337 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:09:57.009609 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:09:57.009612 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:09:57.009614 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:09:57.009803 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:09:57.010144 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:09:57.010151 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:09:57.010477 (+ 326us) client_negotiation.cc:769] Sending connection context
0620 19:09:57.010704 (+ 227us) client_negotiation.cc:240] Negotiation successful
0620 19:09:57.011069 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":106,"threads_started":1}
I20260620 19:09:57.013312 10214 heartbeater.cc:347] Connected to a master server at 127.9.155.126:38609
I20260620 19:09:57.013552 10214 heartbeater.cc:464] Registering TS with master...
I20260620 19:09:57.014036 10214 heartbeater.cc:511] Master 127.9.155.126:38609 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.155.67@KRBTEST.COM" created.
I20260620 19:09:57.015103 9889 ts_manager.cc:194] Registered new tserver with Master: 9eee0c5e3b524a6d8dd9474778b5011a (127.9.155.66:37319)
I20260620 19:09:57.015656 9889 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.155.66@KRBTEST.COM'} at 127.9.155.66:60375
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.155.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.67.keytab.
Entry for principal kudu/127.9.155.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.67.keytab.
I20260620 19:09:57.030920 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.67:0
--local_ip_for_outbound_sockets=127.9.155.67
--webserver_interface=127.9.155.67
--webserver_port=0
--tserver_master_addrs=127.9.155.126:38609
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.67.keytab
--principal=kudu/127.9.155.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:33911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:09:57.140563 10219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:09:57.140836 10219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:09:57.140899 10219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:09:57.144893 10219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:09:57.144953 10219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:09:57.145030 10219 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.67
I20260620 19:09:57.150309 10219 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:33911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.67.keytab
--principal=kudu/127.9.155.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.155.67
--webserver_port=0
--tserver_master_addrs=127.9.155.126:38609
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10219
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.67
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:09:57.151327 10219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:09:57.152326 10219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:09:57.161576 10225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:57.161621 10227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:09:57.161679 10219 server_base.cc:1034] running on GCE node
W20260620 19:09:57.161624 10224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:09:57.162181 10219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:09:57.162809 10219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:09:57.163978 10219 hybrid_clock.cc:648] HybridClock initialized: now 1781982597163956 us; error 33 us; skew 500 ppm
Jun 20 19:09:57 dist-test-slave-n3fx krb5kdc[9844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982597, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:09:57.166661 10219 init.cc:377] Logged in from keytab as kudu/127.9.155.67@KRBTEST.COM (short username kudu)
I20260620 19:09:57.167529 10219 webserver.cc:460] Webserver started at http://127.9.155.67:38061/ using document root <none> and password file <none>
I20260620 19:09:57.168077 10219 fs_manager.cc:359] Metadata directory not provided
I20260620 19:09:57.168138 10219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:09:57.168463 10219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:09:57.170570 10219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/minicluster-data/ts-2/data/instance:
uuid: "0c0d14114e634183a257edc56783c688"
format_stamp: "Formatted at 2026-06-20 19:09:57 on dist-test-slave-n3fx"
server_key: "6d3fe8fd116482e83e7495b9dd784ac4"
server_key_iv: "cce752a07eadb861167d57c32bc417f6"
server_key_version: "encryptionkey@0"
I20260620 19:09:57.171123 10219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/minicluster-data/ts-2/wal/instance:
uuid: "0c0d14114e634183a257edc56783c688"
format_stamp: "Formatted at 2026-06-20 19:09:57 on dist-test-slave-n3fx"
server_key: "6d3fe8fd116482e83e7495b9dd784ac4"
server_key_iv: "cce752a07eadb861167d57c32bc417f6"
server_key_version: "encryptionkey@0"
I20260620 19:09:57.174955 10219 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260620 19:09:57.177604 10234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:57.178893 10219 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 19:09:57.179018 10219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/minicluster-data/ts-2/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/minicluster-data/ts-2/wal
uuid: "0c0d14114e634183a257edc56783c688"
format_stamp: "Formatted at 2026-06-20 19:09:57 on dist-test-slave-n3fx"
server_key: "6d3fe8fd116482e83e7495b9dd784ac4"
server_key_iv: "cce752a07eadb861167d57c32bc417f6"
server_key_version: "encryptionkey@0"
I20260620 19:09:57.179132 10219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:09:57.191010 10219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:09:57.194037 10219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:09:57.194214 10219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:09:57.194821 10219 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:09:57.195652 10219 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:09:57.195696 10219 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:57.195744 10219 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:09:57.195765 10219 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:57.204490 10219 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.67:33297
I20260620 19:09:57.204510 10347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.67:33297 every 8 connection(s)
I20260620 19:09:57.205361 10219 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
I20260620 19:09:57.206949 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 10219
I20260620 19:09:57.207039 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982595231230-9837-0/minicluster-data/ts-2/wal/instance
I20260620 19:09:57.207319 9837 external_mini_cluster.cc:1428] Setting key 4715c2d73b4ea8c2145ebf93f75260ee
Jun 20 19:09:57 dist-test-slave-n3fx krb5kdc[9844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982597, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.67@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:09:57.219116 9928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:57.208618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.67:41221 (local address 127.9.155.126:38609)
0620 19:09:57.208732 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:57.208736 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:57.208751 (+ 15us) server_negotiation.cc:407] Connection header received
0620 19:09:57.209423 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:57.209427 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:57.209479 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:57.209550 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:57.210769 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:57.211299 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:57.212516 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:57.212733 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:57.212821 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:57.215332 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:57.215335 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:57.215337 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:09:57.215370 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:57.216864 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:57.217470 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:57.217473 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:57.217475 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:57.217571 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:57.217974 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:57.217978 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:57.217979 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:57.218181 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:57.218325 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:57.218866 (+ 541us) server_negotiation.cc:299] Negotiation successful
0620 19:09:57.218979 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 19:09:57.220082 10350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:57.207625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:38609 (local address 127.9.155.67:41221)
0620 19:09:57.208363 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:57.208419 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 19:09:57.209219 (+ 800us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:09:57.209741 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:09:57.209749 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:09:57.210134 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:09:57.210622 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:57.210636 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:57.211426 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:57.211430 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:57.212396 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:57.212407 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:57.212915 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:57.212920 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:57.213076 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:57.213628 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:09:57.213644 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:09:57.215158 (+ 1514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:09:57.217029 (+ 1871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:09:57.217035 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:09:57.217049 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:09:57.217339 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:09:57.217689 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:09:57.217692 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:09:57.217693 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:09:57.217853 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:09:57.218345 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:09:57.218351 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:09:57.218612 (+ 261us) client_negotiation.cc:769] Sending connection context
0620 19:09:57.218786 (+ 174us) client_negotiation.cc:240] Negotiation successful
0620 19:09:57.219090 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":119,"threads_started":1}
I20260620 19:09:57.221196 10348 heartbeater.cc:347] Connected to a master server at 127.9.155.126:38609
I20260620 19:09:57.221459 10348 heartbeater.cc:464] Registering TS with master...
I20260620 19:09:57.221998 10348 heartbeater.cc:511] Master 127.9.155.126:38609 requested a full tablet report, sending...
I20260620 19:09:57.223024 9888 ts_manager.cc:194] Registered new tserver with Master: 0c0d14114e634183a257edc56783c688 (127.9.155.67:33297)
I20260620 19:09:57.223623 9888 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.155.67@KRBTEST.COM'} at 127.9.155.67:41221
I20260620 19:09:57.233752 9837 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 19:09:57 dist-test-slave-n3fx krb5kdc[9844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982597, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 19:09:57 dist-test-slave-n3fx krb5kdc[9844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982597, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.155.65@KRBTEST.COM
I20260620 19:09:57.259229 10355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:57.249523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55222 (local address 127.9.155.65:34949)
0620 19:09:57.249856 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:57.249865 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:57.249890 (+ 25us) server_negotiation.cc:407] Connection header received
0620 19:09:57.249961 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:57.249966 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:57.250150 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:57.250278 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:57.250992 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:57.251766 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:57.252866 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:57.253351 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:57.253441 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:57.255900 (+ 2459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:57.255911 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:57.255924 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:09:57.255984 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:57.257649 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:57.258064 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:57.258069 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:57.258070 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:57.258166 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:57.258370 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:57.258372 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:57.258373 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:57.258610 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:57.258786 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:57.258946 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 19:09:57.259075 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":136,"threads_started":1}
W20260620 19:09:57.260313 9994 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55222
Jun 20 19:09:57 dist-test-slave-n3fx krb5kdc[9844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 19:09:57 dist-test-slave-n3fx krb5kdc[9844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.155.65@KRBTEST.COM
I20260620 19:09:57.285534 10355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:57.277608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55232 (local address 127.9.155.65:34949)
0620 19:09:57.277757 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:57.277762 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:57.277893 (+ 131us) server_negotiation.cc:407] Connection header received
0620 19:09:57.278049 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:57.278053 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:57.278100 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:57.278176 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:57.278821 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:57.279392 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:57.280199 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:57.280546 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:57.280627 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:57.282796 (+ 2169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:57.282799 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:57.282801 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:09:57.282826 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:57.284240 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:57.284584 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:57.284586 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:57.284587 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:57.284659 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:57.284839 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:57.284841 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:57.284842 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:57.284990 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:57.285076 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:57.285217 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 19:09:57.285361 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 19:09:57.286530 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 9951
I20260620 19:09:57.291445 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 10085
I20260620 19:09:57.296047 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 10219
I20260620 19:09:57.300443 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 9857
2026-06-20T19:09:57Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2061 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:09:57 dist-test-slave-n3fx krb5kdc[10362](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:09:57 dist-test-slave-n3fx krb5kdc[10362](info): set up 2 sockets
Jun 20 19:09:57 dist-test-slave-n3fx krb5kdc[10362](info): commencing operation
krb5kdc: starting...
W20260620 19:09:58.259814 9837 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.003s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/krb5kdc/test-user.keytab.
Jun 20 19:09:58 dist-test-slave-n3fx krb5kdc[10362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:09:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:09:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.155.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.155.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.155.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab.
Entry for principal kudu/127.9.155.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab.
I20260620 19:09:58.380463 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:44397
--webserver_interface=127.9.155.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab
--principal=kudu/127.9.155.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:40499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.155.126:44397
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:09:58.490435 10375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:09:58.490726 10375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:09:58.490765 10375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:09:58.494839 10375 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:09:58.494915 10375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:09:58.494932 10375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:09:58.494948 10375 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:09:58.494961 10375 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:09:58.500341 10375 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:40499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.155.126:44397
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab
--principal=kudu/127.9.155.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:44397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.155.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10375
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:09:58.501421 10375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:09:58.502435 10375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:09:58.510324 10383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:58.510380 10381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:58.510329 10380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:09:58.510715 10375 server_base.cc:1034] running on GCE node
I20260620 19:09:58.511262 10375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:09:58.512353 10375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:09:58.513515 10375 hybrid_clock.cc:648] HybridClock initialized: now 1781982598513499 us; error 30 us; skew 500 ppm
Jun 20 19:09:58 dist-test-slave-n3fx krb5kdc[10362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982598, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:09:58.520567 10375 init.cc:377] Logged in from keytab as kudu/127.9.155.126@KRBTEST.COM (short username kudu)
I20260620 19:09:58.521584 10375 webserver.cc:460] Webserver started at http://127.9.155.126:44411/ using document root <none> and password file <none>
I20260620 19:09:58.522115 10375 fs_manager.cc:359] Metadata directory not provided
I20260620 19:09:58.522183 10375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:09:58.522368 10375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:09:58.524519 10375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/minicluster-data/master-0/data/instance:
uuid: "b03440abc2c341fda35839b918f3ab06"
format_stamp: "Formatted at 2026-06-20 19:09:58 on dist-test-slave-n3fx"
server_key: "cd9d01b21337b65e17e8b263238bb145"
server_key_iv: "c95b5839287f64073e876ca92da97b5a"
server_key_version: "encryptionkey@0"
I20260620 19:09:58.525053 10375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/minicluster-data/master-0/wal/instance:
uuid: "b03440abc2c341fda35839b918f3ab06"
format_stamp: "Formatted at 2026-06-20 19:09:58 on dist-test-slave-n3fx"
server_key: "cd9d01b21337b65e17e8b263238bb145"
server_key_iv: "c95b5839287f64073e876ca92da97b5a"
server_key_version: "encryptionkey@0"
I20260620 19:09:58.528739 10375 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 19:09:58.531193 10390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:58.532584 10375 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:09:58.532685 10375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/minicluster-data/master-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/minicluster-data/master-0/wal
uuid: "b03440abc2c341fda35839b918f3ab06"
format_stamp: "Formatted at 2026-06-20 19:09:58 on dist-test-slave-n3fx"
server_key: "cd9d01b21337b65e17e8b263238bb145"
server_key_iv: "c95b5839287f64073e876ca92da97b5a"
server_key_version: "encryptionkey@0"
I20260620 19:09:58.532774 10375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:09:58.556095 10375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:09:58.559298 10375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:09:58.559455 10375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:09:58.566097 10375 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.126:44397
I20260620 19:09:58.566118 10442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.126:44397 every 8 connection(s)
I20260620 19:09:58.567027 10375 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
I20260620 19:09:58.569748 10443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:09:58.575147 10443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b03440abc2c341fda35839b918f3ab06: Bootstrap starting.
I20260620 19:09:58.576864 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 10375
I20260620 19:09:58.576970 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/minicluster-data/master-0/wal/instance
I20260620 19:09:58.577219 9837 external_mini_cluster.cc:1428] Setting key e7b72b98391d9c743dc2984909a19b6f
I20260620 19:09:58.577801 10443 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b03440abc2c341fda35839b918f3ab06: Neither blocks nor log segments found. Creating new log.
I20260620 19:09:58.578617 10443 log.cc:824] T 00000000000000000000000000000000 P b03440abc2c341fda35839b918f3ab06: Log is configured to *not* fsync() on all Append() calls
I20260620 19:09:58.580511 10443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b03440abc2c341fda35839b918f3ab06: No bootstrap required, opened a new log
I20260620 19:09:58.584609 10443 raft_consensus.cc:348] T 00000000000000000000000000000000 P b03440abc2c341fda35839b918f3ab06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b03440abc2c341fda35839b918f3ab06" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 44397 } }
I20260620 19:09:58.584885 10443 raft_consensus.cc:374] T 00000000000000000000000000000000 P b03440abc2c341fda35839b918f3ab06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:09:58.584947 10443 raft_consensus.cc:729] T 00000000000000000000000000000000 P b03440abc2c341fda35839b918f3ab06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b03440abc2c341fda35839b918f3ab06, State: Initialized, Role: FOLLOWER
I20260620 19:09:58.585415 10443 consensus_queue.cc:260] T 00000000000000000000000000000000 P b03440abc2c341fda35839b918f3ab06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b03440abc2c341fda35839b918f3ab06" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 44397 } }
I20260620 19:09:58.585551 10443 raft_consensus.cc:388] T 00000000000000000000000000000000 P b03440abc2c341fda35839b918f3ab06 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:09:58.585613 10443 raft_consensus.cc:482] T 00000000000000000000000000000000 P b03440abc2c341fda35839b918f3ab06 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:09:58.585709 10443 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b03440abc2c341fda35839b918f3ab06 [term 0 FOLLOWER]: Advancing to term 1
Jun 20 19:09:58 dist-test-slave-n3fx krb5kdc[10362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:09:58.586630 10443 raft_consensus.cc:504] T 00000000000000000000000000000000 P b03440abc2c341fda35839b918f3ab06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b03440abc2c341fda35839b918f3ab06" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 44397 } }
I20260620 19:09:58.586974 10443 leader_election.cc:302] T 00000000000000000000000000000000 P b03440abc2c341fda35839b918f3ab06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b03440abc2c341fda35839b918f3ab06; no voters:
I20260620 19:09:58.587313 10443 leader_election.cc:288] T 00000000000000000000000000000000 P b03440abc2c341fda35839b918f3ab06 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:09:58.587451 10448 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b03440abc2c341fda35839b918f3ab06 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:09:58.587713 10448 raft_consensus.cc:686] T 00000000000000000000000000000000 P b03440abc2c341fda35839b918f3ab06 [term 1 LEADER]: Becoming Leader. State: Replica: b03440abc2c341fda35839b918f3ab06, State: Running, Role: LEADER
I20260620 19:09:58.588078 10448 consensus_queue.cc:237] T 00000000000000000000000000000000 P b03440abc2c341fda35839b918f3ab06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b03440abc2c341fda35839b918f3ab06" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 44397 } }
I20260620 19:09:58.588821 10443 sys_catalog.cc:564] T 00000000000000000000000000000000 P b03440abc2c341fda35839b918f3ab06 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:09:58.589671 10450 sys_catalog.cc:455] T 00000000000000000000000000000000 P b03440abc2c341fda35839b918f3ab06 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b03440abc2c341fda35839b918f3ab06. Latest consensus state: current_term: 1 leader_uuid: "b03440abc2c341fda35839b918f3ab06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b03440abc2c341fda35839b918f3ab06" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 44397 } } }
I20260620 19:09:58.589808 10450 sys_catalog.cc:458] T 00000000000000000000000000000000 P b03440abc2c341fda35839b918f3ab06 [sys.catalog]: This master's current role is: LEADER
I20260620 19:09:58.590113 10449 sys_catalog.cc:455] T 00000000000000000000000000000000 P b03440abc2c341fda35839b918f3ab06 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b03440abc2c341fda35839b918f3ab06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b03440abc2c341fda35839b918f3ab06" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 44397 } } }
I20260620 19:09:58.590260 10449 sys_catalog.cc:458] T 00000000000000000000000000000000 P b03440abc2c341fda35839b918f3ab06 [sys.catalog]: This master's current role is: LEADER
I20260620 19:09:58.590528 10457 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:09:58.593101 10446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:58.578388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41550 (local address 127.9.155.126:44397)
0620 19:09:58.578844 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:58.578860 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:58.578895 (+ 35us) server_negotiation.cc:407] Connection header received
0620 19:09:58.579686 (+ 791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:58.579707 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:58.580103 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:58.580440 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:58.581198 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:58.582703 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:58.583591 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:58.584322 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:58.584487 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:58.587156 (+ 2669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:58.587163 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:58.587181 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:09:58.587241 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:58.589307 (+ 2066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:58.589853 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:58.589859 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:58.589866 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:58.590010 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:58.590315 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:58.590319 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:58.590321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:58.590873 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:58.591071 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:58.591789 (+ 718us) server_negotiation.cc:299] Negotiation successful
0620 19:09:58.592121 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":131,"threads_started":1}
I20260620 19:09:58.593521 10457 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:09:58.598940 10457 catalog_manager.cc:1347] Generated new cluster ID: e3a8cf268b8c48e298619566442a3bde
I20260620 19:09:58.599026 10457 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:09:58.612886 10457 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:09:58.614187 10457 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:09:58.623600 10457 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b03440abc2c341fda35839b918f3ab06: Generated new TSK 0
I20260620 19:09:58.624380 10457 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.155.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.155.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.155.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.65.keytab.
Entry for principal kudu/127.9.155.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.65.keytab.
I20260620 19:09:58.658610 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.65:0
--local_ip_for_outbound_sockets=127.9.155.65
--webserver_interface=127.9.155.65
--webserver_port=0
--tserver_master_addrs=127.9.155.126:44397
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.65.keytab
--principal=kudu/127.9.155.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:40499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:09:58.768144 10469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:09:58.768440 10469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:09:58.768482 10469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:09:58.772430 10469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:09:58.772488 10469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:09:58.772547 10469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.65
I20260620 19:09:58.777840 10469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:40499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.65.keytab
--principal=kudu/127.9.155.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.155.65
--webserver_port=0
--tserver_master_addrs=127.9.155.126:44397
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.65
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:09:58.778879 10469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:09:58.779821 10469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:09:58.789165 10475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:58.789261 10474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:58.789144 10477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:09:58.789438 10469 server_base.cc:1034] running on GCE node
I20260620 19:09:58.789889 10469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:09:58.790542 10469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:09:58.791705 10469 hybrid_clock.cc:648] HybridClock initialized: now 1781982598791674 us; error 43 us; skew 500 ppm
Jun 20 19:09:58 dist-test-slave-n3fx krb5kdc[10362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982598, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:09:58.794528 10469 init.cc:377] Logged in from keytab as kudu/127.9.155.65@KRBTEST.COM (short username kudu)
I20260620 19:09:58.795439 10469 webserver.cc:460] Webserver started at http://127.9.155.65:36693/ using document root <none> and password file <none>
I20260620 19:09:58.795961 10469 fs_manager.cc:359] Metadata directory not provided
I20260620 19:09:58.796025 10469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:09:58.796253 10469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:09:58.798255 10469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/minicluster-data/ts-0/data/instance:
uuid: "65384d593d564b0ab11dd67af82e111a"
format_stamp: "Formatted at 2026-06-20 19:09:58 on dist-test-slave-n3fx"
server_key: "c9a7981f76f0f8b5ceb53a4a06810481"
server_key_iv: "6313cc91711ab61756463b7650b60c08"
server_key_version: "encryptionkey@0"
I20260620 19:09:58.798802 10469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/minicluster-data/ts-0/wal/instance:
uuid: "65384d593d564b0ab11dd67af82e111a"
format_stamp: "Formatted at 2026-06-20 19:09:58 on dist-test-slave-n3fx"
server_key: "c9a7981f76f0f8b5ceb53a4a06810481"
server_key_iv: "6313cc91711ab61756463b7650b60c08"
server_key_version: "encryptionkey@0"
I20260620 19:09:58.802603 10469 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 19:09:58.805249 10484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:58.806521 10469 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:09:58.806635 10469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/minicluster-data/ts-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/minicluster-data/ts-0/wal
uuid: "65384d593d564b0ab11dd67af82e111a"
format_stamp: "Formatted at 2026-06-20 19:09:58 on dist-test-slave-n3fx"
server_key: "c9a7981f76f0f8b5ceb53a4a06810481"
server_key_iv: "6313cc91711ab61756463b7650b60c08"
server_key_version: "encryptionkey@0"
I20260620 19:09:58.806741 10469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:09:58.835570 10469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:09:58.838812 10469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:09:58.838995 10469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:09:58.839593 10469 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:09:58.840448 10469 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:09:58.840495 10469 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:58.840550 10469 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:09:58.840577 10469 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:58.849804 10469 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.65:35643
I20260620 19:09:58.849848 10597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.65:35643 every 8 connection(s)
I20260620 19:09:58.850734 10469 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
I20260620 19:09:58.854905 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 10469
I20260620 19:09:58.855011 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/minicluster-data/ts-0/wal/instance
I20260620 19:09:58.855300 9837 external_mini_cluster.cc:1428] Setting key e38db2355cdad29fe49f10602cab2eab
Jun 20 19:09:58 dist-test-slave-n3fx krb5kdc[10362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982598, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.65@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.155.66@KRBTEST.COM; defaulting to no policy
I20260620 19:09:58.865794 10446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:58.852650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.65:45709 (local address 127.9.155.126:44397)
0620 19:09:58.852800 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:58.852803 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:58.853719 (+ 916us) server_negotiation.cc:407] Connection header received
0620 19:09:58.854990 (+ 1271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:58.854994 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:58.855050 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:58.855132 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:58.856687 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:58.857244 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:58.858814 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:58.859171 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:58.859393 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:58.861991 (+ 2598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:58.861995 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:58.861999 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:09:58.862032 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:58.863483 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:58.864025 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:58.864029 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:58.864031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:58.864121 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:58.864578 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:58.864582 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:58.864583 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:58.864832 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:58.864933 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:58.865473 (+ 540us) server_negotiation.cc:299] Negotiation successful
0620 19:09:58.865642 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 19:09:58.866671 10600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:58.853024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:44397 (local address 127.9.155.65:45709)
0620 19:09:58.853543 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:58.853588 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 19:09:58.854788 (+ 1200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:09:58.855404 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:09:58.855413 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:09:58.855868 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:09:58.856516 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:58.856536 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:58.857420 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:58.857424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:58.858679 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:58.858689 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:58.859431 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:58.859434 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:58.859586 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:58.860140 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:09:58.860157 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:09:58.861810 (+ 1653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:09:58.863615 (+ 1805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:09:58.863620 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:09:58.863632 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:09:58.863919 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:09:58.864325 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:09:58.864328 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:09:58.864329 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:09:58.864480 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:09:58.864932 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:09:58.864938 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:09:58.865210 (+ 272us) client_negotiation.cc:769] Sending connection context
0620 19:09:58.865379 (+ 169us) client_negotiation.cc:240] Negotiation successful
0620 19:09:58.865678 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":129,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.155.66@KRBTEST.COM" created.
I20260620 19:09:58.867969 10598 heartbeater.cc:347] Connected to a master server at 127.9.155.126:44397
I20260620 19:09:58.868201 10598 heartbeater.cc:464] Registering TS with master...
I20260620 19:09:58.868759 10598 heartbeater.cc:511] Master 127.9.155.126:44397 requested a full tablet report, sending...
I20260620 19:09:58.870271 10407 ts_manager.cc:194] Registered new tserver with Master: 65384d593d564b0ab11dd67af82e111a (127.9.155.65:35643)
I20260620 19:09:58.871464 10407 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.155.65@KRBTEST.COM'} at 127.9.155.65:45709
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.155.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.66.keytab.
Entry for principal kudu/127.9.155.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.66.keytab.
I20260620 19:09:58.881142 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.66:0
--local_ip_for_outbound_sockets=127.9.155.66
--webserver_interface=127.9.155.66
--webserver_port=0
--tserver_master_addrs=127.9.155.126:44397
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.66.keytab
--principal=kudu/127.9.155.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:40499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:09:58.990398 10603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:09:58.990723 10603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:09:58.990795 10603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:09:58.995153 10603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:09:58.995246 10603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:09:58.995337 10603 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.66
I20260620 19:09:59.001010 10603 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:40499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.66.keytab
--principal=kudu/127.9.155.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.155.66
--webserver_port=0
--tserver_master_addrs=127.9.155.126:44397
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10603
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.66
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:09:59.002131 10603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:09:59.003206 10603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:09:59.012871 10609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:59.012879 10608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:09:59.013021 10603 server_base.cc:1034] running on GCE node
W20260620 19:09:59.012933 10611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:09:59.013525 10603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:09:59.014160 10603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:09:59.015336 10603 hybrid_clock.cc:648] HybridClock initialized: now 1781982599015302 us; error 42 us; skew 500 ppm
Jun 20 19:09:59 dist-test-slave-n3fx krb5kdc[10362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982599, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:09:59.018028 10603 init.cc:377] Logged in from keytab as kudu/127.9.155.66@KRBTEST.COM (short username kudu)
I20260620 19:09:59.018844 10603 webserver.cc:460] Webserver started at http://127.9.155.66:42589/ using document root <none> and password file <none>
I20260620 19:09:59.019351 10603 fs_manager.cc:359] Metadata directory not provided
I20260620 19:09:59.019407 10603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:09:59.019572 10603 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:09:59.021608 10603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/minicluster-data/ts-1/data/instance:
uuid: "d7882f831b0e4649a454dec2dcf9722c"
format_stamp: "Formatted at 2026-06-20 19:09:59 on dist-test-slave-n3fx"
server_key: "45dad60ff602eaf80fab4df16aa19d2c"
server_key_iv: "facd7ee988da05a5d0738b532f76470a"
server_key_version: "encryptionkey@0"
I20260620 19:09:59.022118 10603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/minicluster-data/ts-1/wal/instance:
uuid: "d7882f831b0e4649a454dec2dcf9722c"
format_stamp: "Formatted at 2026-06-20 19:09:59 on dist-test-slave-n3fx"
server_key: "45dad60ff602eaf80fab4df16aa19d2c"
server_key_iv: "facd7ee988da05a5d0738b532f76470a"
server_key_version: "encryptionkey@0"
I20260620 19:09:59.025782 10603 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260620 19:09:59.028187 10618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:59.029496 10603 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 19:09:59.029593 10603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/minicluster-data/ts-1/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/minicluster-data/ts-1/wal
uuid: "d7882f831b0e4649a454dec2dcf9722c"
format_stamp: "Formatted at 2026-06-20 19:09:59 on dist-test-slave-n3fx"
server_key: "45dad60ff602eaf80fab4df16aa19d2c"
server_key_iv: "facd7ee988da05a5d0738b532f76470a"
server_key_version: "encryptionkey@0"
I20260620 19:09:59.029687 10603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:09:59.047178 10603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:09:59.050351 10603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:09:59.050511 10603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:09:59.051085 10603 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:09:59.051883 10603 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:09:59.051929 10603 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:59.051970 10603 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:09:59.051983 10603 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:59.061039 10603 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.66:42997
I20260620 19:09:59.061071 10731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.66:42997 every 8 connection(s)
I20260620 19:09:59.062031 10603 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
I20260620 19:09:59.067572 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 10603
I20260620 19:09:59.067677 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/minicluster-data/ts-1/wal/instance
I20260620 19:09:59.067921 9837 external_mini_cluster.cc:1428] Setting key 6ff0fc25dc28c0d2258167db408bb706
Jun 20 19:09:59 dist-test-slave-n3fx krb5kdc[10362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982599, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.66@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:09:59.075659 10446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:59.063881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.66:36189 (local address 127.9.155.126:44397)
0620 19:09:59.064060 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:59.064063 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:59.064941 (+ 878us) server_negotiation.cc:407] Connection header received
0620 19:09:59.066272 (+ 1331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:59.066276 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:59.066336 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:59.066414 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:59.067629 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:59.068257 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:59.069539 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:59.069767 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:59.069853 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:59.072311 (+ 2458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:59.072314 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:59.072316 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:09:59.072342 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:59.073675 (+ 1333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:59.074171 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:59.074174 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:59.074175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:59.074251 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:59.074563 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:59.074567 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:59.074568 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:59.074754 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:59.074845 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:59.075361 (+ 516us) server_negotiation.cc:299] Negotiation successful
0620 19:09:59.075493 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
WARNING: no policy specified for kudu/127.9.155.67@KRBTEST.COM; defaulting to no policy
I20260620 19:09:59.076623 10734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:59.064271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:44397 (local address 127.9.155.66:36189)
0620 19:09:59.064776 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:59.064824 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 19:09:59.066038 (+ 1214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:09:59.066595 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:09:59.066603 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:09:59.066963 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:09:59.067455 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:59.067470 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:59.068381 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:59.068384 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:59.069425 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:59.069436 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:59.069908 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:59.069912 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:59.070075 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:59.070663 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:09:59.070679 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:09:59.072124 (+ 1445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:09:59.073785 (+ 1661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:09:59.073792 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:09:59.073808 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:09:59.074077 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:09:59.074331 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:09:59.074334 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:09:59.074336 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:09:59.074479 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:09:59.074866 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:09:59.074873 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:09:59.075127 (+ 254us) client_negotiation.cc:769] Sending connection context
0620 19:09:59.075297 (+ 170us) client_negotiation.cc:240] Negotiation successful
0620 19:09:59.075637 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":123,"threads_started":1}
I20260620 19:09:59.077792 10732 heartbeater.cc:347] Connected to a master server at 127.9.155.126:44397
I20260620 19:09:59.078030 10732 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.155.67@KRBTEST.COM" created.
I20260620 19:09:59.078552 10732 heartbeater.cc:511] Master 127.9.155.126:44397 requested a full tablet report, sending...
I20260620 19:09:59.079684 10407 ts_manager.cc:194] Registered new tserver with Master: d7882f831b0e4649a454dec2dcf9722c (127.9.155.66:42997)
I20260620 19:09:59.080343 10407 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.155.66@KRBTEST.COM'} at 127.9.155.66:36189
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.155.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.67.keytab.
Entry for principal kudu/127.9.155.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.67.keytab.
I20260620 19:09:59.092811 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.67:0
--local_ip_for_outbound_sockets=127.9.155.67
--webserver_interface=127.9.155.67
--webserver_port=0
--tserver_master_addrs=127.9.155.126:44397
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.67.keytab
--principal=kudu/127.9.155.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:40499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:09:59.200989 10737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:09:59.201273 10737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:09:59.201316 10737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:09:59.205343 10737 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:09:59.205415 10737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:09:59.205476 10737 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.67
I20260620 19:09:59.210916 10737 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:40499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.67.keytab
--principal=kudu/127.9.155.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.155.67
--webserver_port=0
--tserver_master_addrs=127.9.155.126:44397
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10737
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.67
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:09:59.211980 10737 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:09:59.212971 10737 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:09:59.222796 10742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:59.222783 10743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:09:59.222783 10745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:09:59.223153 10737 server_base.cc:1034] running on GCE node
I20260620 19:09:59.223570 10737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:09:59.224272 10737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:09:59.225445 10737 hybrid_clock.cc:648] HybridClock initialized: now 1781982599225428 us; error 34 us; skew 500 ppm
Jun 20 19:09:59 dist-test-slave-n3fx krb5kdc[10362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982599, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:09:59.228029 10737 init.cc:377] Logged in from keytab as kudu/127.9.155.67@KRBTEST.COM (short username kudu)
I20260620 19:09:59.228919 10737 webserver.cc:460] Webserver started at http://127.9.155.67:44581/ using document root <none> and password file <none>
I20260620 19:09:59.229435 10737 fs_manager.cc:359] Metadata directory not provided
I20260620 19:09:59.229497 10737 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:09:59.229704 10737 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:09:59.231705 10737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/minicluster-data/ts-2/data/instance:
uuid: "c2ecfde73b614bd6a464befc22642906"
format_stamp: "Formatted at 2026-06-20 19:09:59 on dist-test-slave-n3fx"
server_key: "2104964abe188e1cef88e46b2d51d02a"
server_key_iv: "8ddd43527fcdf0fad575bd49ab00a712"
server_key_version: "encryptionkey@0"
I20260620 19:09:59.232290 10737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/minicluster-data/ts-2/wal/instance:
uuid: "c2ecfde73b614bd6a464befc22642906"
format_stamp: "Formatted at 2026-06-20 19:09:59 on dist-test-slave-n3fx"
server_key: "2104964abe188e1cef88e46b2d51d02a"
server_key_iv: "8ddd43527fcdf0fad575bd49ab00a712"
server_key_version: "encryptionkey@0"
I20260620 19:09:59.236182 10737 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260620 19:09:59.238762 10752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:59.240010 10737 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 19:09:59.240127 10737 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/minicluster-data/ts-2/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/minicluster-data/ts-2/wal
uuid: "c2ecfde73b614bd6a464befc22642906"
format_stamp: "Formatted at 2026-06-20 19:09:59 on dist-test-slave-n3fx"
server_key: "2104964abe188e1cef88e46b2d51d02a"
server_key_iv: "8ddd43527fcdf0fad575bd49ab00a712"
server_key_version: "encryptionkey@0"
I20260620 19:09:59.240289 10737 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:09:59.256942 10737 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:09:59.260064 10737 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:09:59.260298 10737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:09:59.260893 10737 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:09:59.261716 10737 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:09:59.261765 10737 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:59.261819 10737 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:09:59.261848 10737 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:09:59.270602 10737 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.67:39305
I20260620 19:09:59.270630 10865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.67:39305 every 8 connection(s)
I20260620 19:09:59.271603 10737 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
I20260620 19:09:59.278856 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 10737
I20260620 19:09:59.278982 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982595231230-9837-0/minicluster-data/ts-2/wal/instance
I20260620 19:09:59.279256 9837 external_mini_cluster.cc:1428] Setting key 0b2ebc609432a436c5a2ce41077bfa00
Jun 20 19:09:59 dist-test-slave-n3fx krb5kdc[10362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982599, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.67@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:09:59.285492 10446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:59.273507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.67:50551 (local address 127.9.155.126:44397)
0620 19:09:59.273647 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:59.273652 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:59.274534 (+ 882us) server_negotiation.cc:407] Connection header received
0620 19:09:59.275776 (+ 1242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:59.275779 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:59.275828 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:59.275903 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:59.277155 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:59.277701 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:59.278884 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:59.279267 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:59.279390 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:59.282170 (+ 2780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:59.282174 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:59.282176 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:09:59.282200 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:59.283521 (+ 1321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:59.284045 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:59.284047 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:59.284049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:59.284132 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:59.284490 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:59.284492 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:59.284493 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:59.284670 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:59.284742 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:59.285233 (+ 491us) server_negotiation.cc:299] Negotiation successful
0620 19:09:59.285374 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 19:09:59.286226 10868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:59.273926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:44397 (local address 127.9.155.67:50551)
0620 19:09:59.274370 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:59.274419 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 19:09:59.275544 (+ 1125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:09:59.276050 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:09:59.276057 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:09:59.276433 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:09:59.277009 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:59.277026 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:59.277839 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:59.277842 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:59.278743 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:59.278754 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:59.279452 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:59.279456 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:59.279646 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:59.280354 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:09:59.280374 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:09:59.282017 (+ 1643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:09:59.283650 (+ 1633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:09:59.283655 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:09:59.283668 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:09:59.283959 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:09:59.284255 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:09:59.284260 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:09:59.284262 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:09:59.284408 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:09:59.284750 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:09:59.284767 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:09:59.285001 (+ 234us) client_negotiation.cc:769] Sending connection context
0620 19:09:59.285148 (+ 147us) client_negotiation.cc:240] Negotiation successful
0620 19:09:59.285421 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":114,"threads_started":1}
I20260620 19:09:59.287297 10866 heartbeater.cc:347] Connected to a master server at 127.9.155.126:44397
I20260620 19:09:59.287505 10866 heartbeater.cc:464] Registering TS with master...
I20260620 19:09:59.287947 10866 heartbeater.cc:511] Master 127.9.155.126:44397 requested a full tablet report, sending...
I20260620 19:09:59.288992 10407 ts_manager.cc:194] Registered new tserver with Master: c2ecfde73b614bd6a464befc22642906 (127.9.155.67:39305)
I20260620 19:09:59.289649 10407 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.155.67@KRBTEST.COM'} at 127.9.155.67:50551
I20260620 19:09:59.293247 9837 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:09:59.303404 10446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:59.296507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41566 (local address 127.9.155.126:44397)
0620 19:09:59.296647 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:59.296652 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:59.296772 (+ 120us) server_negotiation.cc:407] Connection header received
0620 19:09:59.296909 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:59.296911 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:59.296954 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:59.297026 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:59.297656 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:59.298196 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:59.298992 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:59.299199 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:59.299276 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:59.300533 (+ 1257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:59.300536 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:59.300538 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:09:59.300560 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:59.301885 (+ 1325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:59.302292 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:59.302295 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:59.302296 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:59.302376 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:59.302651 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:59.302653 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:59.302654 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:59.302862 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:59.302965 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:59.303164 (+ 199us) server_negotiation.cc:299] Negotiation successful
0620 19:09:59.303250 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 19:09:59.308338 10407 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41566:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:09:59.311296 10407 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:09:59.325361 10879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:59.319676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33956 (local address 127.9.155.66:42997)
0620 19:09:59.320651 (+ 975us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:59.320664 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:59.320710 (+ 46us) server_negotiation.cc:407] Connection header received
0620 19:09:59.320793 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:59.320798 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:59.321249 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:59.321431 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:09:59.321658 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:59.322408 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:59.323862 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:59.324681 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:59.324791 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:59.324944 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:59.325106 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 19:09:59.325200 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":872,"thread_start_us":134,"threads_started":1}
I20260620 19:09:59.325356 10880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:59.319541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.66:42997 (local address 127.0.0.1:33956)
0620 19:09:59.320168 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:59.320239 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0620 19:09:59.320373 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:09:59.321420 (+ 1047us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:09:59.321423 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:09:59.321443 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:09:59.321551 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:59.321558 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:59.322527 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:59.322531 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:59.323726 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:59.323737 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:59.324803 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:59.324812 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:59.324930 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:59.324969 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 19:09:59.325069 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 19:09:59.325200 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"spinlock_wait_cycles":5760,"thread_start_us":78,"threads_started":1}
I20260620 19:09:59.325747 10878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:59.319815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.65:35643 (local address 127.0.0.1:47456)
0620 19:09:59.320879 (+ 1064us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:59.320891 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:09:59.320981 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:09:59.321554 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:09:59.321557 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:09:59.321574 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:09:59.321646 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:59.321651 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:59.323085 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:59.323088 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:59.324101 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:59.324109 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:59.325398 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:59.325401 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:59.325473 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:59.325497 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 19:09:59.325564 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 19:09:59.325621 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1007,"mutex_wait_us":27,"thread_start_us":84,"threads_started":1}
I20260620 19:09:59.325842 10881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:59.320339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47456 (local address 127.9.155.65:35643)
0620 19:09:59.320733 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:59.320746 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:59.320968 (+ 222us) server_negotiation.cc:407] Connection header received
0620 19:09:59.321076 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:59.321081 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:59.321257 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:59.321402 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:09:59.321760 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:59.322923 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:59.324251 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:59.325078 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:59.325182 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:59.325332 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:59.325606 (+ 274us) server_negotiation.cc:299] Negotiation successful
0620 19:09:59.325708 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":88,"threads_started":1}
I20260620 19:09:59.326123 10877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:59.319605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.67:39305 (local address 127.0.0.1:42170)
0620 19:09:59.320168 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:59.320238 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0620 19:09:59.320373 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:09:59.321238 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:09:59.321244 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:09:59.321265 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:09:59.321432 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:59.321441 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:59.323054 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:59.323058 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:59.324252 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:59.324264 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:59.325226 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:59.325230 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:59.325316 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:59.325337 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 19:09:59.325386 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 19:09:59.326029 (+ 643us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"mutex_wait_us":1,"spinlock_wait_cycles":4864,"thread_start_us":106,"threads_started":1}
I20260620 19:09:59.326807 10882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:59.320322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42170 (local address 127.9.155.67:39305)
0620 19:09:59.320753 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:59.320765 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:59.320789 (+ 24us) server_negotiation.cc:407] Connection header received
0620 19:09:59.320864 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:59.320869 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:59.321068 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:59.321592 (+ 524us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:09:59.321749 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:59.322923 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:59.324365 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:59.325107 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:59.326378 (+ 1271us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:59.326517 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:59.326566 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 19:09:59.326654 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":125,"threads_started":1}
I20260620 19:09:59.328558 10666 tablet_service.cc:1467] Processing CreateTablet for tablet cf0081bdd3c84d939a5d986cbdcc8c55 (DEFAULT_TABLE table=test-table [id=bf2524b2614441c9a3b762b46d700047]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:09:59.328572 10532 tablet_service.cc:1467] Processing CreateTablet for tablet cf0081bdd3c84d939a5d986cbdcc8c55 (DEFAULT_TABLE table=test-table [id=bf2524b2614441c9a3b762b46d700047]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:09:59.329469 10666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf0081bdd3c84d939a5d986cbdcc8c55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:09:59.329574 10532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf0081bdd3c84d939a5d986cbdcc8c55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:09:59.329353 10800 tablet_service.cc:1467] Processing CreateTablet for tablet cf0081bdd3c84d939a5d986cbdcc8c55 (DEFAULT_TABLE table=test-table [id=bf2524b2614441c9a3b762b46d700047]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:09:59.330245 10800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf0081bdd3c84d939a5d986cbdcc8c55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:09:59.335633 10883 tablet_bootstrap.cc:492] T cf0081bdd3c84d939a5d986cbdcc8c55 P d7882f831b0e4649a454dec2dcf9722c: Bootstrap starting.
I20260620 19:09:59.337549 10885 tablet_bootstrap.cc:492] T cf0081bdd3c84d939a5d986cbdcc8c55 P c2ecfde73b614bd6a464befc22642906: Bootstrap starting.
I20260620 19:09:59.338696 10883 tablet_bootstrap.cc:654] T cf0081bdd3c84d939a5d986cbdcc8c55 P d7882f831b0e4649a454dec2dcf9722c: Neither blocks nor log segments found. Creating new log.
I20260620 19:09:59.339385 10883 log.cc:824] T cf0081bdd3c84d939a5d986cbdcc8c55 P d7882f831b0e4649a454dec2dcf9722c: Log is configured to *not* fsync() on all Append() calls
I20260620 19:09:59.339566 10885 tablet_bootstrap.cc:654] T cf0081bdd3c84d939a5d986cbdcc8c55 P c2ecfde73b614bd6a464befc22642906: Neither blocks nor log segments found. Creating new log.
I20260620 19:09:59.340198 10885 log.cc:824] T cf0081bdd3c84d939a5d986cbdcc8c55 P c2ecfde73b614bd6a464befc22642906: Log is configured to *not* fsync() on all Append() calls
I20260620 19:09:59.340596 10884 tablet_bootstrap.cc:492] T cf0081bdd3c84d939a5d986cbdcc8c55 P 65384d593d564b0ab11dd67af82e111a: Bootstrap starting.
I20260620 19:09:59.341233 10883 tablet_bootstrap.cc:492] T cf0081bdd3c84d939a5d986cbdcc8c55 P d7882f831b0e4649a454dec2dcf9722c: No bootstrap required, opened a new log
I20260620 19:09:59.341416 10883 ts_tablet_manager.cc:1397] T cf0081bdd3c84d939a5d986cbdcc8c55 P d7882f831b0e4649a454dec2dcf9722c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 19:09:59.342155 10885 tablet_bootstrap.cc:492] T cf0081bdd3c84d939a5d986cbdcc8c55 P c2ecfde73b614bd6a464befc22642906: No bootstrap required, opened a new log
I20260620 19:09:59.342334 10885 ts_tablet_manager.cc:1397] T cf0081bdd3c84d939a5d986cbdcc8c55 P c2ecfde73b614bd6a464befc22642906: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260620 19:09:59.342877 10884 tablet_bootstrap.cc:654] T cf0081bdd3c84d939a5d986cbdcc8c55 P 65384d593d564b0ab11dd67af82e111a: Neither blocks nor log segments found. Creating new log.
I20260620 19:09:59.344116 10884 log.cc:824] T cf0081bdd3c84d939a5d986cbdcc8c55 P 65384d593d564b0ab11dd67af82e111a: Log is configured to *not* fsync() on all Append() calls
I20260620 19:09:59.344951 10883 raft_consensus.cc:348] T cf0081bdd3c84d939a5d986cbdcc8c55 P d7882f831b0e4649a454dec2dcf9722c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7882f831b0e4649a454dec2dcf9722c" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 42997 } } peers { permanent_uuid: "c2ecfde73b614bd6a464befc22642906" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 39305 } } peers { permanent_uuid: "65384d593d564b0ab11dd67af82e111a" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 35643 } }
I20260620 19:09:59.345275 10883 raft_consensus.cc:374] T cf0081bdd3c84d939a5d986cbdcc8c55 P d7882f831b0e4649a454dec2dcf9722c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:09:59.345342 10883 raft_consensus.cc:729] T cf0081bdd3c84d939a5d986cbdcc8c55 P d7882f831b0e4649a454dec2dcf9722c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7882f831b0e4649a454dec2dcf9722c, State: Initialized, Role: FOLLOWER
I20260620 19:09:59.345798 10883 consensus_queue.cc:260] T cf0081bdd3c84d939a5d986cbdcc8c55 P d7882f831b0e4649a454dec2dcf9722c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7882f831b0e4649a454dec2dcf9722c" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 42997 } } peers { permanent_uuid: "c2ecfde73b614bd6a464befc22642906" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 39305 } } peers { permanent_uuid: "65384d593d564b0ab11dd67af82e111a" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 35643 } }
I20260620 19:09:59.346467 10732 heartbeater.cc:503] Master 127.9.155.126:44397 was elected leader, sending a full tablet report...
I20260620 19:09:59.346508 10884 tablet_bootstrap.cc:492] T cf0081bdd3c84d939a5d986cbdcc8c55 P 65384d593d564b0ab11dd67af82e111a: No bootstrap required, opened a new log
I20260620 19:09:59.346678 10884 ts_tablet_manager.cc:1397] T cf0081bdd3c84d939a5d986cbdcc8c55 P 65384d593d564b0ab11dd67af82e111a: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260620 19:09:59.346807 10883 ts_tablet_manager.cc:1428] T cf0081bdd3c84d939a5d986cbdcc8c55 P d7882f831b0e4649a454dec2dcf9722c: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 19:09:59.347177 10885 raft_consensus.cc:348] T cf0081bdd3c84d939a5d986cbdcc8c55 P c2ecfde73b614bd6a464befc22642906 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7882f831b0e4649a454dec2dcf9722c" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 42997 } } peers { permanent_uuid: "c2ecfde73b614bd6a464befc22642906" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 39305 } } peers { permanent_uuid: "65384d593d564b0ab11dd67af82e111a" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 35643 } }
I20260620 19:09:59.347390 10885 raft_consensus.cc:374] T cf0081bdd3c84d939a5d986cbdcc8c55 P c2ecfde73b614bd6a464befc22642906 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:09:59.347448 10885 raft_consensus.cc:729] T cf0081bdd3c84d939a5d986cbdcc8c55 P c2ecfde73b614bd6a464befc22642906 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ecfde73b614bd6a464befc22642906, State: Initialized, Role: FOLLOWER
I20260620 19:09:59.347894 10885 consensus_queue.cc:260] T cf0081bdd3c84d939a5d986cbdcc8c55 P c2ecfde73b614bd6a464befc22642906 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7882f831b0e4649a454dec2dcf9722c" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 42997 } } peers { permanent_uuid: "c2ecfde73b614bd6a464befc22642906" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 39305 } } peers { permanent_uuid: "65384d593d564b0ab11dd67af82e111a" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 35643 } }
I20260620 19:09:59.348578 10866 heartbeater.cc:503] Master 127.9.155.126:44397 was elected leader, sending a full tablet report...
I20260620 19:09:59.348912 10885 ts_tablet_manager.cc:1428] T cf0081bdd3c84d939a5d986cbdcc8c55 P c2ecfde73b614bd6a464befc22642906: Time spent starting tablet: real 0.007s user 0.000s sys 0.006s
I20260620 19:09:59.350272 10884 raft_consensus.cc:348] T cf0081bdd3c84d939a5d986cbdcc8c55 P 65384d593d564b0ab11dd67af82e111a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7882f831b0e4649a454dec2dcf9722c" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 42997 } } peers { permanent_uuid: "c2ecfde73b614bd6a464befc22642906" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 39305 } } peers { permanent_uuid: "65384d593d564b0ab11dd67af82e111a" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 35643 } }
I20260620 19:09:59.350512 10884 raft_consensus.cc:374] T cf0081bdd3c84d939a5d986cbdcc8c55 P 65384d593d564b0ab11dd67af82e111a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:09:59.350571 10884 raft_consensus.cc:729] T cf0081bdd3c84d939a5d986cbdcc8c55 P 65384d593d564b0ab11dd67af82e111a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65384d593d564b0ab11dd67af82e111a, State: Initialized, Role: FOLLOWER
I20260620 19:09:59.351039 10884 consensus_queue.cc:260] T cf0081bdd3c84d939a5d986cbdcc8c55 P 65384d593d564b0ab11dd67af82e111a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7882f831b0e4649a454dec2dcf9722c" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 42997 } } peers { permanent_uuid: "c2ecfde73b614bd6a464befc22642906" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 39305 } } peers { permanent_uuid: "65384d593d564b0ab11dd67af82e111a" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 35643 } }
I20260620 19:09:59.351687 10598 heartbeater.cc:503] Master 127.9.155.126:44397 was elected leader, sending a full tablet report...
I20260620 19:09:59.352265 10884 ts_tablet_manager.cc:1428] T cf0081bdd3c84d939a5d986cbdcc8c55 P 65384d593d564b0ab11dd67af82e111a: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
W20260620 19:09:59.522568 10867 tablet.cc:2369] T cf0081bdd3c84d939a5d986cbdcc8c55 P c2ecfde73b614bd6a464befc22642906: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:09:59.533423 10889 raft_consensus.cc:482] T cf0081bdd3c84d939a5d986cbdcc8c55 P d7882f831b0e4649a454dec2dcf9722c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:09:59.533617 10889 raft_consensus.cc:504] T cf0081bdd3c84d939a5d986cbdcc8c55 P d7882f831b0e4649a454dec2dcf9722c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7882f831b0e4649a454dec2dcf9722c" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 42997 } } peers { permanent_uuid: "c2ecfde73b614bd6a464befc22642906" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 39305 } } peers { permanent_uuid: "65384d593d564b0ab11dd67af82e111a" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 35643 } }
I20260620 19:09:59.534873 10889 leader_election.cc:288] T cf0081bdd3c84d939a5d986cbdcc8c55 P d7882f831b0e4649a454dec2dcf9722c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2ecfde73b614bd6a464befc22642906 (127.9.155.67:39305), 65384d593d564b0ab11dd67af82e111a (127.9.155.65:35643)
I20260620 19:09:59.538822 10734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:59.535110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.67:39305 (local address 127.9.155.66:57707)
0620 19:09:59.535220 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:59.535235 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 19:09:59.535333 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:09:59.535639 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:09:59.535642 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:09:59.535661 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:09:59.535798 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:59.535804 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:59.536587 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:59.536590 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:59.537577 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:59.537587 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:59.538341 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:59.538345 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:59.538458 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:59.538485 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 19:09:59.538577 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 19:09:59.538653 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":21}
I20260620 19:09:59.538862 10882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:59.535329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.66:57707 (local address 127.9.155.67:39305)
0620 19:09:59.535442 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:59.535447 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:59.535459 (+ 12us) server_negotiation.cc:407] Connection header received
0620 19:09:59.535506 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:59.535508 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:59.535547 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:59.535610 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:09:59.535911 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:59.536481 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:59.537710 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:59.538215 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:59.538322 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:59.538378 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:59.538647 (+ 269us) server_negotiation.cc:299] Negotiation successful
0620 19:09:59.538735 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 19:09:59.539081 10881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:59.535284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.66:41031 (local address 127.9.155.65:35643)
0620 19:09:59.535408 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:59.535413 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:59.535671 (+ 258us) server_negotiation.cc:407] Connection header received
0620 19:09:59.535804 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:59.535807 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:59.535849 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:59.535913 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:09:59.536231 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:59.536865 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:59.538040 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:59.538518 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:59.538625 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:59.538671 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:59.538887 (+ 216us) server_negotiation.cc:299] Negotiation successful
0620 19:09:59.538963 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 19:09:59.539085 10892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:59.535180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.65:35643 (local address 127.9.155.66:41031)
0620 19:09:59.535564 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0620 19:09:59.535575 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:09:59.535692 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:09:59.535959 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:09:59.535961 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:09:59.535977 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:09:59.536085 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:59.536090 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:59.536969 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:59.536971 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:59.537906 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:09:59.537917 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:09:59.538648 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:09:59.538652 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:09:59.538753 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:59.538776 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 19:09:59.538859 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 19:09:59.538954 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"mutex_wait_us":1,"thread_start_us":131,"threads_started":1}
I20260620 19:09:59.539736 10820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf0081bdd3c84d939a5d986cbdcc8c55" candidate_uuid: "d7882f831b0e4649a454dec2dcf9722c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ecfde73b614bd6a464befc22642906" is_pre_election: true
I20260620 19:09:59.539733 10552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf0081bdd3c84d939a5d986cbdcc8c55" candidate_uuid: "d7882f831b0e4649a454dec2dcf9722c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65384d593d564b0ab11dd67af82e111a" is_pre_election: true
I20260620 19:09:59.540023 10820 raft_consensus.cc:2454] T cf0081bdd3c84d939a5d986cbdcc8c55 P c2ecfde73b614bd6a464befc22642906 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7882f831b0e4649a454dec2dcf9722c in term 0.
I20260620 19:09:59.540027 10552 raft_consensus.cc:2454] T cf0081bdd3c84d939a5d986cbdcc8c55 P 65384d593d564b0ab11dd67af82e111a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7882f831b0e4649a454dec2dcf9722c in term 0.
I20260620 19:09:59.540508 10622 leader_election.cc:302] T cf0081bdd3c84d939a5d986cbdcc8c55 P d7882f831b0e4649a454dec2dcf9722c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65384d593d564b0ab11dd67af82e111a, d7882f831b0e4649a454dec2dcf9722c; no voters:
I20260620 19:09:59.540808 10889 raft_consensus.cc:2781] T cf0081bdd3c84d939a5d986cbdcc8c55 P d7882f831b0e4649a454dec2dcf9722c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:09:59.540933 10889 raft_consensus.cc:482] T cf0081bdd3c84d939a5d986cbdcc8c55 P d7882f831b0e4649a454dec2dcf9722c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:09:59.540982 10889 raft_consensus.cc:3037] T cf0081bdd3c84d939a5d986cbdcc8c55 P d7882f831b0e4649a454dec2dcf9722c [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:09:59.542024 10889 raft_consensus.cc:504] T cf0081bdd3c84d939a5d986cbdcc8c55 P d7882f831b0e4649a454dec2dcf9722c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7882f831b0e4649a454dec2dcf9722c" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 42997 } } peers { permanent_uuid: "c2ecfde73b614bd6a464befc22642906" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 39305 } } peers { permanent_uuid: "65384d593d564b0ab11dd67af82e111a" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 35643 } }
I20260620 19:09:59.542351 10889 leader_election.cc:288] T cf0081bdd3c84d939a5d986cbdcc8c55 P d7882f831b0e4649a454dec2dcf9722c [CANDIDATE]: Term 1 election: Requested vote from peers c2ecfde73b614bd6a464befc22642906 (127.9.155.67:39305), 65384d593d564b0ab11dd67af82e111a (127.9.155.65:35643)
I20260620 19:09:59.542793 10552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf0081bdd3c84d939a5d986cbdcc8c55" candidate_uuid: "d7882f831b0e4649a454dec2dcf9722c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65384d593d564b0ab11dd67af82e111a"
I20260620 19:09:59.542923 10552 raft_consensus.cc:3037] T cf0081bdd3c84d939a5d986cbdcc8c55 P 65384d593d564b0ab11dd67af82e111a [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:09:59.544049 10552 raft_consensus.cc:2454] T cf0081bdd3c84d939a5d986cbdcc8c55 P 65384d593d564b0ab11dd67af82e111a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7882f831b0e4649a454dec2dcf9722c in term 1.
I20260620 19:09:59.544422 10622 leader_election.cc:302] T cf0081bdd3c84d939a5d986cbdcc8c55 P d7882f831b0e4649a454dec2dcf9722c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65384d593d564b0ab11dd67af82e111a, d7882f831b0e4649a454dec2dcf9722c; no voters:
I20260620 19:09:59.544397 10820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf0081bdd3c84d939a5d986cbdcc8c55" candidate_uuid: "d7882f831b0e4649a454dec2dcf9722c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ecfde73b614bd6a464befc22642906"
I20260620 19:09:59.544535 10820 raft_consensus.cc:3037] T cf0081bdd3c84d939a5d986cbdcc8c55 P c2ecfde73b614bd6a464befc22642906 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:09:59.545549 10820 raft_consensus.cc:2454] T cf0081bdd3c84d939a5d986cbdcc8c55 P c2ecfde73b614bd6a464befc22642906 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7882f831b0e4649a454dec2dcf9722c in term 1.
I20260620 19:09:59.545753 10889 raft_consensus.cc:2781] T cf0081bdd3c84d939a5d986cbdcc8c55 P d7882f831b0e4649a454dec2dcf9722c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:09:59.545990 10889 raft_consensus.cc:686] T cf0081bdd3c84d939a5d986cbdcc8c55 P d7882f831b0e4649a454dec2dcf9722c [term 1 LEADER]: Becoming Leader. State: Replica: d7882f831b0e4649a454dec2dcf9722c, State: Running, Role: LEADER
I20260620 19:09:59.546373 10889 consensus_queue.cc:237] T cf0081bdd3c84d939a5d986cbdcc8c55 P d7882f831b0e4649a454dec2dcf9722c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7882f831b0e4649a454dec2dcf9722c" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 42997 } } peers { permanent_uuid: "c2ecfde73b614bd6a464befc22642906" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 39305 } } peers { permanent_uuid: "65384d593d564b0ab11dd67af82e111a" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 35643 } }
I20260620 19:09:59.549471 10406 catalog_manager.cc:5495] T cf0081bdd3c84d939a5d986cbdcc8c55 P d7882f831b0e4649a454dec2dcf9722c reported cstate change: term changed from 0 to 1, leader changed from <none> to d7882f831b0e4649a454dec2dcf9722c (127.9.155.66). New cstate: current_term: 1 leader_uuid: "d7882f831b0e4649a454dec2dcf9722c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7882f831b0e4649a454dec2dcf9722c" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 42997 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2ecfde73b614bd6a464befc22642906" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 39305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65384d593d564b0ab11dd67af82e111a" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 35643 } health_report { overall_health: UNKNOWN } } }
Jun 20 19:09:59 dist-test-slave-n3fx krb5kdc[10362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.155.65@KRBTEST.COM
I20260620 19:09:59.563134 10881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:59.554416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47468 (local address 127.9.155.65:35643)
0620 19:09:59.554562 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:59.554566 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:59.554679 (+ 113us) server_negotiation.cc:407] Connection header received
0620 19:09:59.554785 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:59.554788 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:59.554832 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:59.554899 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:59.555626 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:59.556118 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:59.557072 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:59.557280 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:59.557349 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:59.559725 (+ 2376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:59.559737 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:59.559751 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:09:59.559776 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:59.561346 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:59.561765 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:59.561771 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:59.561772 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:59.561874 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:59.562196 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:59.562199 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:59.562201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:59.562502 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:59.562677 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:59.562865 (+ 188us) server_negotiation.cc:299] Negotiation successful
0620 19:09:59.562979 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260620 19:09:59.563115 10733 tablet.cc:2369] T cf0081bdd3c84d939a5d986cbdcc8c55 P d7882f831b0e4649a454dec2dcf9722c: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 19:09:59 dist-test-slave-n3fx krb5kdc[10362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982599, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 19:09:59 dist-test-slave-n3fx krb5kdc[10362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982599, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.155.65@KRBTEST.COM
I20260620 19:09:59.590171 10881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:59.581757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47470 (local address 127.9.155.65:35643)
0620 19:09:59.581917 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:59.581922 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:59.582010 (+ 88us) server_negotiation.cc:407] Connection header received
0620 19:09:59.582160 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:59.582164 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:59.582206 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:59.582347 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:59.583034 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:59.583544 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:59.584416 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:59.584632 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:59.584719 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:59.587088 (+ 2369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:59.587093 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:59.587095 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:09:59.587128 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:59.588511 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:59.588931 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:59.588935 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:59.588937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:59.589022 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:59.589269 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:59.589272 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:59.589274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:59.589468 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:59.589582 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:59.589896 (+ 314us) server_negotiation.cc:299] Negotiation successful
0620 19:09:59.590015 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
Jun 20 19:09:59 dist-test-slave-n3fx krb5kdc[10362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
W20260620 19:09:59.602183 10599 tablet.cc:2369] T cf0081bdd3c84d939a5d986cbdcc8c55 P 65384d593d564b0ab11dd67af82e111a: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 19:09:59 dist-test-slave-n3fx krb5kdc[10362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.155.65@KRBTEST.COM
I20260620 19:09:59.615566 10881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:09:59.607079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47472 (local address 127.9.155.65:35643)
0620 19:09:59.607233 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 19:09:59.607238 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:09:59.607412 (+ 174us) server_negotiation.cc:407] Connection header received
0620 19:09:59.607560 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:09:59.607562 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:09:59.607605 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:09:59.607669 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:09:59.608377 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:59.608872 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:59.609698 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:09:59.609914 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:09:59.610000 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:09:59.612549 (+ 2549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:09:59.612552 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:09:59.612555 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:09:59.612582 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:09:59.613928 (+ 1346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:59.614412 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:59.614416 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:59.614417 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:59.614490 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:09:59.614756 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:09:59.614759 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:09:59.614760 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:09:59.614934 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:09:59.615046 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 19:09:59.615297 (+ 251us) server_negotiation.cc:299] Negotiation successful
0620 19:09:59.615409 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260620 19:09:59.616982 10512 tablet_service.cc:2908] Rejecting scan request for tablet cf0081bdd3c84d939a5d986cbdcc8c55: Uninitialized: clean time has not yet been initialized
I20260620 19:09:59.618816 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 10469
I20260620 19:09:59.623780 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 10603
I20260620 19:09:59.628759 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 10737
I20260620 19:09:59.633610 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 10375
2026-06-20T19:09:59Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2333 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:09:59 dist-test-slave-n3fx krb5kdc[10907](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:09:59 dist-test-slave-n3fx krb5kdc[10907](info): set up 2 sockets
Jun 20 19:09:59 dist-test-slave-n3fx krb5kdc[10907](info): commencing operation
krb5kdc: starting...
W20260620 19:10:00.591787 9837 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/krb5kdc/test-user.keytab.
Jun 20 19:10:00 dist-test-slave-n3fx krb5kdc[10907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:10:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:10:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.155.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.155.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.155.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab.
Entry for principal kudu/127.9.155.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab.
I20260620 19:10:00.711162 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:43665
--webserver_interface=127.9.155.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab
--principal=kudu/127.9.155.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:36397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.155.126:43665
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:10:00.819806 10920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:00.820099 10920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:00.820158 10920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:00.824313 10920 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:10:00.824393 10920 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 19:10:00.824416 10920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:00.824440 10920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:00.824461 10920 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:10:00.824488 10920 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:10:00.830050 10920 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:36397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.155.126:43665
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab
--principal=kudu/127.9.155.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:43665
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.155.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10920
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:10:00.831141 10920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:00.832145 10920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:10:00.840678 10928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:00.840678 10926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:00.840678 10925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:00.840875 10920 server_base.cc:1034] running on GCE node
I20260620 19:10:00.841552 10920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:00.842685 10920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:00.843883 10920 hybrid_clock.cc:648] HybridClock initialized: now 1781982600843867 us; error 36 us; skew 500 ppm
Jun 20 19:10:00 dist-test-slave-n3fx krb5kdc[10907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982600, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:00.846778 10920 init.cc:377] Logged in from keytab as kudu/127.9.155.126@KRBTEST.COM (short username kudu)
I20260620 19:10:00.847730 10920 webserver.cc:460] Webserver started at http://127.9.155.126:39055/ using document root <none> and password file <none>
I20260620 19:10:00.848333 10920 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:00.848409 10920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:00.848614 10920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:10:00.850701 10920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/minicluster-data/master-0/data/instance:
uuid: "05b0609251bd43e5aeb2a0b1a38f4f49"
format_stamp: "Formatted at 2026-06-20 19:10:00 on dist-test-slave-n3fx"
server_key: "37eaa643d03d1ded3b24200e26de66bb"
server_key_iv: "19347d5d87f736cee46811cb9343860f"
server_key_version: "encryptionkey@0"
I20260620 19:10:00.851248 10920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/minicluster-data/master-0/wal/instance:
uuid: "05b0609251bd43e5aeb2a0b1a38f4f49"
format_stamp: "Formatted at 2026-06-20 19:10:00 on dist-test-slave-n3fx"
server_key: "37eaa643d03d1ded3b24200e26de66bb"
server_key_iv: "19347d5d87f736cee46811cb9343860f"
server_key_version: "encryptionkey@0"
I20260620 19:10:00.855732 10920 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260620 19:10:00.858304 10935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:00.859508 10920 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 19:10:00.859614 10920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/minicluster-data/master-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/minicluster-data/master-0/wal
uuid: "05b0609251bd43e5aeb2a0b1a38f4f49"
format_stamp: "Formatted at 2026-06-20 19:10:00 on dist-test-slave-n3fx"
server_key: "37eaa643d03d1ded3b24200e26de66bb"
server_key_iv: "19347d5d87f736cee46811cb9343860f"
server_key_version: "encryptionkey@0"
I20260620 19:10:00.859722 10920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:00.872354 10920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:00.876070 10920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:00.876318 10920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:00.884312 10920 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.126:43665
I20260620 19:10:00.884341 10997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.126:43665 every 8 connection(s)
I20260620 19:10:00.885317 10920 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
I20260620 19:10:00.887116 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 10920
I20260620 19:10:00.887291 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/minicluster-data/master-0/wal/instance
I20260620 19:10:00.887543 9837 external_mini_cluster.cc:1428] Setting key 1dc08c69fa1737c7110e0a240cf44c91
I20260620 19:10:00.888475 10998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:00.895462 10998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05b0609251bd43e5aeb2a0b1a38f4f49: Bootstrap starting.
Jun 20 19:10:00 dist-test-slave-n3fx krb5kdc[10907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:10:00.898288 10998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05b0609251bd43e5aeb2a0b1a38f4f49: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:00.899626 10998 log.cc:824] T 00000000000000000000000000000000 P 05b0609251bd43e5aeb2a0b1a38f4f49: Log is configured to *not* fsync() on all Append() calls
I20260620 19:10:00.902190 10998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05b0609251bd43e5aeb2a0b1a38f4f49: No bootstrap required, opened a new log
I20260620 19:10:00.903360 11001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:00.889061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50810 (local address 127.9.155.126:43665)
0620 19:10:00.889492 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:00.889510 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:00.889552 (+ 42us) server_negotiation.cc:407] Connection header received
0620 19:10:00.890388 (+ 836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:00.890413 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:00.890845 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:00.891232 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:00.891892 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:00.893415 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:00.894328 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:00.895025 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:00.895184 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:00.898030 (+ 2846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:00.898037 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:00.898055 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:00.898092 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:00.900097 (+ 2005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:00.900660 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:00.900667 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:00.900674 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:00.900823 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:00.901111 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:00.901115 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:00.901116 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:00.901636 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:00.901831 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:00.902063 (+ 232us) server_negotiation.cc:299] Negotiation successful
0620 19:10:00.902403 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":118,"threads_started":1}
I20260620 19:10:00.905913 10998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05b0609251bd43e5aeb2a0b1a38f4f49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05b0609251bd43e5aeb2a0b1a38f4f49" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 43665 } }
I20260620 19:10:00.906139 10998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 05b0609251bd43e5aeb2a0b1a38f4f49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:00.906173 10998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05b0609251bd43e5aeb2a0b1a38f4f49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05b0609251bd43e5aeb2a0b1a38f4f49, State: Initialized, Role: FOLLOWER
I20260620 19:10:00.906587 10998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05b0609251bd43e5aeb2a0b1a38f4f49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05b0609251bd43e5aeb2a0b1a38f4f49" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 43665 } }
I20260620 19:10:00.906759 10998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 05b0609251bd43e5aeb2a0b1a38f4f49 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:10:00.906831 10998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05b0609251bd43e5aeb2a0b1a38f4f49 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:10:00.906956 10998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05b0609251bd43e5aeb2a0b1a38f4f49 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:00.907859 10998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05b0609251bd43e5aeb2a0b1a38f4f49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05b0609251bd43e5aeb2a0b1a38f4f49" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 43665 } }
I20260620 19:10:00.908243 10998 leader_election.cc:302] T 00000000000000000000000000000000 P 05b0609251bd43e5aeb2a0b1a38f4f49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05b0609251bd43e5aeb2a0b1a38f4f49; no voters:
I20260620 19:10:00.908588 10998 leader_election.cc:288] T 00000000000000000000000000000000 P 05b0609251bd43e5aeb2a0b1a38f4f49 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:10:00.908687 11003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05b0609251bd43e5aeb2a0b1a38f4f49 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:10:00.908843 11003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05b0609251bd43e5aeb2a0b1a38f4f49 [term 1 LEADER]: Becoming Leader. State: Replica: 05b0609251bd43e5aeb2a0b1a38f4f49, State: Running, Role: LEADER
I20260620 19:10:00.909148 11003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05b0609251bd43e5aeb2a0b1a38f4f49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05b0609251bd43e5aeb2a0b1a38f4f49" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 43665 } }
I20260620 19:10:00.909900 10998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05b0609251bd43e5aeb2a0b1a38f4f49 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:10:00.910333 11003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05b0609251bd43e5aeb2a0b1a38f4f49 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05b0609251bd43e5aeb2a0b1a38f4f49. Latest consensus state: current_term: 1 leader_uuid: "05b0609251bd43e5aeb2a0b1a38f4f49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05b0609251bd43e5aeb2a0b1a38f4f49" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 43665 } } }
I20260620 19:10:00.910447 11003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05b0609251bd43e5aeb2a0b1a38f4f49 [sys.catalog]: This master's current role is: LEADER
I20260620 19:10:00.910821 11007 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:10:00.910791 11004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05b0609251bd43e5aeb2a0b1a38f4f49 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "05b0609251bd43e5aeb2a0b1a38f4f49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05b0609251bd43e5aeb2a0b1a38f4f49" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 43665 } } }
I20260620 19:10:00.910885 11004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05b0609251bd43e5aeb2a0b1a38f4f49 [sys.catalog]: This master's current role is: LEADER
I20260620 19:10:00.914016 11007 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:10:00.919958 11007 catalog_manager.cc:1347] Generated new cluster ID: 370ad1d73b954ae1a0b83b6ca95b79ff
I20260620 19:10:00.920045 11007 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:10:00.938256 11007 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:10:00.939270 11007 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:10:00.951489 11007 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 05b0609251bd43e5aeb2a0b1a38f4f49: Generated new TSK 0
I20260620 19:10:00.952286 11007 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.155.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.155.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.155.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.65.keytab.
Entry for principal kudu/127.9.155.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.65.keytab.
I20260620 19:10:00.983424 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.65:0
--local_ip_for_outbound_sockets=127.9.155.65
--webserver_interface=127.9.155.65
--webserver_port=0
--tserver_master_addrs=127.9.155.126:43665
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.65.keytab
--principal=kudu/127.9.155.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:36397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:10:01.103018 11024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:01.103317 11024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:01.103364 11024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:01.107717 11024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:01.107808 11024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:01.107851 11024 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 19:10:01.107884 11024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.65
I20260620 19:10:01.113602 11024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:36397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.65.keytab
--principal=kudu/127.9.155.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.155.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.155.126:43665
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.65
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:10:01.114708 11024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:01.115729 11024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:10:01.124967 11029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:01.124969 11030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:01.125010 11032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:01.125316 11024 server_base.cc:1034] running on GCE node
I20260620 19:10:01.125681 11024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:01.126284 11024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:01.127460 11024 hybrid_clock.cc:648] HybridClock initialized: now 1781982601127430 us; error 41 us; skew 500 ppm
Jun 20 19:10:01 dist-test-slave-n3fx krb5kdc[10907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982601, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:01.130457 11024 init.cc:377] Logged in from keytab as kudu/127.9.155.65@KRBTEST.COM (short username kudu)
I20260620 19:10:01.131479 11024 webserver.cc:460] Webserver started at http://127.9.155.65:36039/ using document root <none> and password file <none>
I20260620 19:10:01.132014 11024 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:01.132077 11024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:01.132301 11024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:10:01.134364 11024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/minicluster-data/ts-0/data/instance:
uuid: "3964b77bd96f435cab098d389d3b20f1"
format_stamp: "Formatted at 2026-06-20 19:10:01 on dist-test-slave-n3fx"
server_key: "9809e605809308f8524488023aa046ac"
server_key_iv: "5a4dc063e925227b22e443383fe2e35f"
server_key_version: "encryptionkey@0"
I20260620 19:10:01.134913 11024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/minicluster-data/ts-0/wal/instance:
uuid: "3964b77bd96f435cab098d389d3b20f1"
format_stamp: "Formatted at 2026-06-20 19:10:01 on dist-test-slave-n3fx"
server_key: "9809e605809308f8524488023aa046ac"
server_key_iv: "5a4dc063e925227b22e443383fe2e35f"
server_key_version: "encryptionkey@0"
I20260620 19:10:01.138496 11024 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 19:10:01.140978 11039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:01.142288 11024 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 19:10:01.142402 11024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/minicluster-data/ts-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/minicluster-data/ts-0/wal
uuid: "3964b77bd96f435cab098d389d3b20f1"
format_stamp: "Formatted at 2026-06-20 19:10:01 on dist-test-slave-n3fx"
server_key: "9809e605809308f8524488023aa046ac"
server_key_iv: "5a4dc063e925227b22e443383fe2e35f"
server_key_version: "encryptionkey@0"
I20260620 19:10:01.142504 11024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:01.177577 11024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:01.180819 11024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:01.181001 11024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:01.182489 11024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:10:01.182543 11024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:01.182600 11024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:10:01.182626 11024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 19:10:01 dist-test-slave-n3fx krb5kdc[10907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982601, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.65@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:10:01.192785 11024 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.65:34885
I20260620 19:10:01.192942 11154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.65:34885 every 8 connection(s)
I20260620 19:10:01.193908 11024 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
I20260620 19:10:01.196456 11001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:01.183406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.65:33103 (local address 127.9.155.126:43665)
0620 19:10:01.183520 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:01.183524 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:01.184649 (+ 1125us) server_negotiation.cc:407] Connection header received
0620 19:10:01.185670 (+ 1021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:01.185675 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:01.185746 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:01.185844 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:01.187272 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:01.188099 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:01.189355 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:01.189580 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:01.189667 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:01.192395 (+ 2728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:01.192399 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:01.192402 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:01.192433 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:01.194171 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:01.194829 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:01.194833 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:01.194835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:01.194928 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:01.195320 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:01.195324 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:01.195325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:01.195527 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:01.195613 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:01.196109 (+ 496us) server_negotiation.cc:299] Negotiation successful
0620 19:10:01.196320 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 19:10:01.197186 11048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:01.183923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:43665 (local address 127.9.155.65:33103)
0620 19:10:01.184503 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:01.184547 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:01.185424 (+ 877us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:01.186030 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:01.186040 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:01.186447 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:01.187103 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:01.187123 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:01.188259 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:01.188263 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:01.189239 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:01.189249 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:01.189718 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:01.189723 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:01.189890 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:01.190548 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:01.190563 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:01.192178 (+ 1615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:01.194363 (+ 2185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:01.194369 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:01.194385 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:01.194708 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:01.195048 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:01.195051 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:01.195052 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:01.195189 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:01.195628 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:01.195634 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:01.195878 (+ 244us) client_negotiation.cc:769] Sending connection context
0620 19:10:01.196027 (+ 149us) client_negotiation.cc:240] Negotiation successful
0620 19:10:01.196372 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":137,"threads_started":1}
I20260620 19:10:01.198431 11155 heartbeater.cc:347] Connected to a master server at 127.9.155.126:43665
I20260620 19:10:01.198567 11155 heartbeater.cc:464] Registering TS with master...
I20260620 19:10:01.199129 11155 heartbeater.cc:511] Master 127.9.155.126:43665 requested a full tablet report, sending...
I20260620 19:10:01.200752 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 11024
I20260620 19:10:01.200734 10951 ts_manager.cc:194] Registered new tserver with Master: 3964b77bd96f435cab098d389d3b20f1 (127.9.155.65:34885)
I20260620 19:10:01.200858 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/minicluster-data/ts-0/wal/instance
I20260620 19:10:01.201104 9837 external_mini_cluster.cc:1428] Setting key b223cc2faab922d2786ea228108a6c86
I20260620 19:10:01.202055 10951 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.155.65@KRBTEST.COM'} at 127.9.155.65:33103
I20260620 19:10:01.208945 11001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:01.200605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.65:39177 (local address 127.9.155.126:43665)
0620 19:10:01.200730 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:01.200734 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:01.201810 (+ 1076us) server_negotiation.cc:407] Connection header received
0620 19:10:01.201923 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:01.201925 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:01.201969 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:01.202051 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:01.202535 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:01.203187 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:01.204124 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:01.204469 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:01.204539 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:01.205787 (+ 1248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:01.205790 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:01.205792 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:01.205813 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:01.207189 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:01.207689 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:01.207693 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:01.207694 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:01.207788 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:01.208088 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:01.208091 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:01.208092 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:01.208354 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:01.208443 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:01.208703 (+ 260us) server_negotiation.cc:299] Negotiation successful
0620 19:10:01.208816 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 19:10:01.209044 11161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:01.201424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:43665 (local address 127.9.155.65:39177)
0620 19:10:01.201730 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:01.201745 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:01.201847 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:01.202080 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:01.202083 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:01.202289 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:01.202400 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:01.202408 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:01.203308 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:01.203311 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:01.203979 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:01.203990 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:01.204686 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:01.204690 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:01.204780 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:01.205250 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:01.205263 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:01.205662 (+ 399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:01.207296 (+ 1634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:01.207299 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:01.207301 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:01.207544 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:01.207879 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:01.207883 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:01.207884 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:01.207982 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:01.208451 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:01.208455 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:01.208557 (+ 102us) client_negotiation.cc:769] Sending connection context
0620 19:10:01.208670 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 19:10:01.208834 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":34,"thread_start_us":89,"threads_started":1}
WARNING: no policy specified for kudu/127.9.155.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.155.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.155.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.66.keytab.
Entry for principal kudu/127.9.155.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.66.keytab.
I20260620 19:10:01.227934 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.66:0
--local_ip_for_outbound_sockets=127.9.155.66
--webserver_interface=127.9.155.66
--webserver_port=0
--tserver_master_addrs=127.9.155.126:43665
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.66.keytab
--principal=kudu/127.9.155.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:36397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:10:01.336478 11164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:01.336771 11164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:01.336835 11164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:01.340975 11164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:01.341050 11164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:01.341104 11164 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 19:10:01.341156 11164 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.66
I20260620 19:10:01.347025 11164 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:36397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.66.keytab
--principal=kudu/127.9.155.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.155.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.155.126:43665
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11164
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.66
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:10:01.348163 11164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:01.349223 11164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:10:01.358888 11172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:01.358892 11169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:01.358888 11170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:01.359294 11164 server_base.cc:1034] running on GCE node
I20260620 19:10:01.359732 11164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:01.360489 11164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:01.361671 11164 hybrid_clock.cc:648] HybridClock initialized: now 1781982601361644 us; error 38 us; skew 500 ppm
Jun 20 19:10:01 dist-test-slave-n3fx krb5kdc[10907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982601, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:01.364789 11164 init.cc:377] Logged in from keytab as kudu/127.9.155.66@KRBTEST.COM (short username kudu)
I20260620 19:10:01.365767 11164 webserver.cc:460] Webserver started at http://127.9.155.66:36943/ using document root <none> and password file <none>
I20260620 19:10:01.366343 11164 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:01.366411 11164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:01.366618 11164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:10:01.368794 11164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/minicluster-data/ts-1/data/instance:
uuid: "72317321300c441f8fb701add3d21f06"
format_stamp: "Formatted at 2026-06-20 19:10:01 on dist-test-slave-n3fx"
server_key: "5271725a05129a42f1bfc2c51ef5fb39"
server_key_iv: "7e9b1042fcf412e9b191adad6e84ecfd"
server_key_version: "encryptionkey@0"
I20260620 19:10:01.369390 11164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/minicluster-data/ts-1/wal/instance:
uuid: "72317321300c441f8fb701add3d21f06"
format_stamp: "Formatted at 2026-06-20 19:10:01 on dist-test-slave-n3fx"
server_key: "5271725a05129a42f1bfc2c51ef5fb39"
server_key_iv: "7e9b1042fcf412e9b191adad6e84ecfd"
server_key_version: "encryptionkey@0"
I20260620 19:10:01.373478 11164 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 19:10:01.376438 11179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:01.377801 11164 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:10:01.377962 11164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/minicluster-data/ts-1/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/minicluster-data/ts-1/wal
uuid: "72317321300c441f8fb701add3d21f06"
format_stamp: "Formatted at 2026-06-20 19:10:01 on dist-test-slave-n3fx"
server_key: "5271725a05129a42f1bfc2c51ef5fb39"
server_key_iv: "7e9b1042fcf412e9b191adad6e84ecfd"
server_key_version: "encryptionkey@0"
I20260620 19:10:01.378082 11164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:01.393702 11164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:01.397009 11164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:01.397202 11164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:01.398816 11164 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:10:01.398870 11164 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:01.398926 11164 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:10:01.398952 11164 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 19:10:01 dist-test-slave-n3fx krb5kdc[10907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982601, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.66@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:10:01.408989 11164 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.66:35937
I20260620 19:10:01.409017 11294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.66:35937 every 8 connection(s)
I20260620 19:10:01.409914 11164 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
I20260620 19:10:01.413185 11001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:01.399643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.66:51961 (local address 127.9.155.126:43665)
0620 19:10:01.399789 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:01.399794 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:01.400759 (+ 965us) server_negotiation.cc:407] Connection header received
0620 19:10:01.402091 (+ 1332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:01.402095 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:01.402155 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:01.402260 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:01.403873 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:01.404825 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:01.406175 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:01.406414 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:01.406506 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:01.409453 (+ 2947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:01.409456 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:01.409459 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:01.409486 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:01.411065 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:01.411613 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:01.411617 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:01.411619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:01.411726 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:01.412138 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:01.412141 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:01.412142 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:01.412372 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:01.412451 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:01.412943 (+ 492us) server_negotiation.cc:299] Negotiation successful
0620 19:10:01.413062 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 19:10:01.414055 11188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:01.400083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:43665 (local address 127.9.155.66:51961)
0620 19:10:01.400599 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:01.400654 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:01.401814 (+ 1160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:01.402468 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:01.402477 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:01.402963 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:01.403686 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:01.403709 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:01.404951 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:01.404955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:01.406049 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:01.406059 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:01.406559 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:01.406563 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:01.406745 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:01.407363 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:01.407388 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:01.409260 (+ 1872us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:01.411227 (+ 1967us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:01.411233 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:01.411248 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:01.411514 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:01.411882 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:01.411885 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:01.411886 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:01.412059 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:01.412458 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:01.412463 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:01.412698 (+ 235us) client_negotiation.cc:769] Sending connection context
0620 19:10:01.412857 (+ 159us) client_negotiation.cc:240] Negotiation successful
0620 19:10:01.413116 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":142,"threads_started":1}
I20260620 19:10:01.414119 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 11164
I20260620 19:10:01.414206 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/minicluster-data/ts-1/wal/instance
I20260620 19:10:01.414474 9837 external_mini_cluster.cc:1428] Setting key 785b58702f38b068db95e8ef34dfd113
I20260620 19:10:01.415941 11295 heartbeater.cc:347] Connected to a master server at 127.9.155.126:43665
I20260620 19:10:01.416108 11295 heartbeater.cc:464] Registering TS with master...
I20260620 19:10:01.416711 11295 heartbeater.cc:511] Master 127.9.155.126:43665 requested a full tablet report, sending...
I20260620 19:10:01.417837 10951 ts_manager.cc:194] Registered new tserver with Master: 72317321300c441f8fb701add3d21f06 (127.9.155.66:35937)
I20260620 19:10:01.418537 10951 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.155.66@KRBTEST.COM'} at 127.9.155.66:51961
WARNING: no policy specified for kudu/127.9.155.67@KRBTEST.COM; defaulting to no policy
I20260620 19:10:01.425829 11001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:01.418250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.66:33747 (local address 127.9.155.126:43665)
0620 19:10:01.418359 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:01.418364 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:01.418592 (+ 228us) server_negotiation.cc:407] Connection header received
0620 19:10:01.418721 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:01.418724 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:01.418766 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:01.418832 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:01.419392 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:01.419903 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:01.420799 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:01.421009 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:01.421093 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:01.422444 (+ 1351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:01.422448 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:01.422450 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:01.422473 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:01.424041 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:01.424583 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:01.424587 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:01.424589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:01.424663 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:01.424931 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:01.424934 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:01.424936 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:01.425152 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:01.425235 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:01.425546 (+ 311us) server_negotiation.cc:299] Negotiation successful
0620 19:10:01.425648 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 19:10:01.425841 11302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:01.418184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:43665 (local address 127.9.155.66:33747)
0620 19:10:01.418514 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:01.418526 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:01.418617 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:01.418937 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:01.418940 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:01.419158 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:01.419283 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:01.419290 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:01.420017 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:01.420021 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:01.420692 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:01.420702 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:01.421103 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:01.421106 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:01.421194 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:01.421811 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:01.421827 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:01.422241 (+ 414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:01.424142 (+ 1901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:01.424146 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:01.424148 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:01.424425 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:01.424748 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:01.424751 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:01.424753 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:01.424841 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:01.425279 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:01.425282 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:01.425376 (+ 94us) client_negotiation.cc:769] Sending connection context
0620 19:10:01.425478 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 19:10:01.425648 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":9,"thread_start_us":110,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.155.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.155.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.67.keytab.
Entry for principal kudu/127.9.155.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.67.keytab.
I20260620 19:10:01.442955 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.67:0
--local_ip_for_outbound_sockets=127.9.155.67
--webserver_interface=127.9.155.67
--webserver_port=0
--tserver_master_addrs=127.9.155.126:43665
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.67.keytab
--principal=kudu/127.9.155.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:36397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:10:01.553213 11304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:01.553506 11304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:01.553551 11304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:01.557636 11304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:01.557713 11304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:01.557753 11304 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 19:10:01.557787 11304 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.67
I20260620 19:10:01.563397 11304 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:36397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.67.keytab
--principal=kudu/127.9.155.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.155.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.155.126:43665
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11304
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.67
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:10:01.564549 11304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:01.565539 11304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:10:01.575078 11312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:01.575093 11310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:01.575081 11309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:01.575464 11304 server_base.cc:1034] running on GCE node
I20260620 19:10:01.575865 11304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:01.576571 11304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:01.577764 11304 hybrid_clock.cc:648] HybridClock initialized: now 1781982601577743 us; error 34 us; skew 500 ppm
Jun 20 19:10:01 dist-test-slave-n3fx krb5kdc[10907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982601, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:01.580816 11304 init.cc:377] Logged in from keytab as kudu/127.9.155.67@KRBTEST.COM (short username kudu)
I20260620 19:10:01.581816 11304 webserver.cc:460] Webserver started at http://127.9.155.67:35643/ using document root <none> and password file <none>
I20260620 19:10:01.582393 11304 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:01.582460 11304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:01.582656 11304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:10:01.584736 11304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/minicluster-data/ts-2/data/instance:
uuid: "0a8b3c70c2ec47589f5869aabbc0ce15"
format_stamp: "Formatted at 2026-06-20 19:10:01 on dist-test-slave-n3fx"
server_key: "11cf1748e968018140246c3ef9c4b478"
server_key_iv: "936560b79721d7ec441ce7ce3c69d270"
server_key_version: "encryptionkey@0"
I20260620 19:10:01.585304 11304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/minicluster-data/ts-2/wal/instance:
uuid: "0a8b3c70c2ec47589f5869aabbc0ce15"
format_stamp: "Formatted at 2026-06-20 19:10:01 on dist-test-slave-n3fx"
server_key: "11cf1748e968018140246c3ef9c4b478"
server_key_iv: "936560b79721d7ec441ce7ce3c69d270"
server_key_version: "encryptionkey@0"
I20260620 19:10:01.589237 11304 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 19:10:01.591907 11319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:01.593344 11304 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:10:01.593488 11304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/minicluster-data/ts-2/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/minicluster-data/ts-2/wal
uuid: "0a8b3c70c2ec47589f5869aabbc0ce15"
format_stamp: "Formatted at 2026-06-20 19:10:01 on dist-test-slave-n3fx"
server_key: "11cf1748e968018140246c3ef9c4b478"
server_key_iv: "936560b79721d7ec441ce7ce3c69d270"
server_key_version: "encryptionkey@0"
I20260620 19:10:01.593604 11304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:01.609161 11304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:01.612360 11304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:01.612537 11304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:01.613969 11304 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:10:01.614024 11304 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:01.614076 11304 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:10:01.614118 11304 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 19:10:01 dist-test-slave-n3fx krb5kdc[10907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982601, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.67@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:10:01.623894 11304 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.67:44381
I20260620 19:10:01.624099 11434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.67:44381 every 8 connection(s)
I20260620 19:10:01.624949 11304 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
I20260620 19:10:01.627990 11001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:01.614841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.67:56769 (local address 127.9.155.126:43665)
0620 19:10:01.614961 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:01.614965 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:01.616159 (+ 1194us) server_negotiation.cc:407] Connection header received
0620 19:10:01.617355 (+ 1196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:01.617360 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:01.617432 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:01.617533 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:01.619143 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:01.619953 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:01.621301 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:01.621638 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:01.621755 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:01.624507 (+ 2752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:01.624510 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:01.624513 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:01.624543 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:01.626088 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:01.626588 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:01.626590 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:01.626592 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:01.626665 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:01.627039 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:01.627041 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:01.627042 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:01.627214 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:01.627320 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:01.627766 (+ 446us) server_negotiation.cc:299] Negotiation successful
0620 19:10:01.627868 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 19:10:01.628787 11328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:01.615419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:43665 (local address 127.9.155.67:56769)
0620 19:10:01.615964 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:01.616011 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:01.617081 (+ 1070us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:01.617737 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:01.617747 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:01.618216 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:01.618943 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:01.618965 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:01.620115 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:01.620118 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:01.621174 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:01.621186 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:01.621799 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:01.621802 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:01.621956 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:01.622626 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:01.622650 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:01.624333 (+ 1683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:01.626221 (+ 1888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:01.626226 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:01.626237 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:01.626493 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:01.626765 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:01.626768 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:01.626770 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:01.626914 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:01.627320 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:01.627325 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:01.627549 (+ 224us) client_negotiation.cc:769] Sending connection context
0620 19:10:01.627702 (+ 153us) client_negotiation.cc:240] Negotiation successful
0620 19:10:01.627972 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":154,"threads_started":1}
I20260620 19:10:01.629213 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 11304
I20260620 19:10:01.629315 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/minicluster-data/ts-2/wal/instance
I20260620 19:10:01.629591 9837 external_mini_cluster.cc:1428] Setting key 3be53d62c3422bab6a0e4614d3ee9e52
I20260620 19:10:01.630136 11435 heartbeater.cc:347] Connected to a master server at 127.9.155.126:43665
I20260620 19:10:01.630282 11435 heartbeater.cc:464] Registering TS with master...
I20260620 19:10:01.630831 11435 heartbeater.cc:511] Master 127.9.155.126:43665 requested a full tablet report, sending...
I20260620 19:10:01.631759 10951 ts_manager.cc:194] Registered new tserver with Master: 0a8b3c70c2ec47589f5869aabbc0ce15 (127.9.155.67:44381)
I20260620 19:10:01.632477 10951 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.155.67@KRBTEST.COM'} at 127.9.155.67:56769
I20260620 19:10:01.639463 11001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:01.632321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.67:41379 (local address 127.9.155.126:43665)
0620 19:10:01.632451 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:01.632455 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:01.632570 (+ 115us) server_negotiation.cc:407] Connection header received
0620 19:10:01.632671 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:01.632674 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:01.632750 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:01.632830 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:01.633283 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:01.633775 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:01.634800 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:01.635013 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:01.635093 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:01.636201 (+ 1108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:01.636204 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:01.636206 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:01.636338 (+ 132us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:01.637666 (+ 1328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:01.638144 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:01.638147 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:01.638148 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:01.638283 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:01.638549 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:01.638552 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:01.638553 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:01.638758 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:01.638832 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:01.639188 (+ 356us) server_negotiation.cc:299] Negotiation successful
0620 19:10:01.639291 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 19:10:01.639712 11441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:01.632156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:43665 (local address 127.9.155.67:41379)
0620 19:10:01.632484 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:01.632498 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:01.632598 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:01.632847 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:01.632850 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:01.633036 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:01.633158 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:01.633165 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:01.633898 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:01.633901 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:01.634698 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:01.634707 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:01.635107 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:01.635109 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:01.635174 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:01.635726 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:01.635740 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:01.636060 (+ 320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:01.637778 (+ 1718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:01.637781 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:01.637785 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:01.638050 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:01.638382 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:01.638385 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:01.638386 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:01.638469 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:01.638869 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:01.638872 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:01.638988 (+ 116us) client_negotiation.cc:769] Sending connection context
0620 19:10:01.639097 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 19:10:01.639355 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":154,"threads_started":1}
I20260620 19:10:01.643376 9837 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 19:10:01 dist-test-slave-n3fx krb5kdc[10907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982601, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 19:10:01 dist-test-slave-n3fx krb5kdc[10907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982601, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:10:01.668751 11001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:01.660371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50826 (local address 127.9.155.126:43665)
0620 19:10:01.660522 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:01.660525 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:01.660540 (+ 15us) server_negotiation.cc:407] Connection header received
0620 19:10:01.660581 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:01.660584 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:01.660623 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:01.660712 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:01.661450 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:01.661967 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:01.662867 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:01.663084 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:01.663171 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:01.665697 (+ 2526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:01.665701 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:01.665704 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:01.665731 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:01.667127 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:01.667551 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:01.667554 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:01.667556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:01.667634 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:01.667883 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:01.667886 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:01.667888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:01.668067 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:01.668189 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:01.668475 (+ 286us) server_negotiation.cc:299] Negotiation successful
0620 19:10:01.668596 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 19:10:01.671984 10951 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50826:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:10:01.674999 10951 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:10:01.689870 11457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:01.684770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37370 (local address 127.9.155.65:34885)
0620 19:10:01.685154 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:01.685169 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:01.685192 (+ 23us) server_negotiation.cc:407] Connection header received
0620 19:10:01.685275 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:01.685280 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:01.685466 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:01.685609 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:10:01.686020 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:01.686757 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:01.688601 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:01.689157 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:01.689254 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:01.689433 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:01.689592 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 19:10:01.689692 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":75,"threads_started":1}
I20260620 19:10:01.689860 11454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:01.683700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.66:35937 (local address 127.0.0.1:48172)
0620 19:10:01.684535 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:01.684548 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:01.684624 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:01.685705 (+ 1081us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:01.685708 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:01.685726 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:10:01.685869 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:01.685877 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:01.687171 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:01.687175 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:01.688158 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:01.688168 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:01.689376 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:01.689380 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:01.689480 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:01.689510 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 19:10:01.689556 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 19:10:01.689716 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":776,"mutex_wait_us":31,"spinlock_wait_cycles":12928,"thread_start_us":113,"threads_started":1}
I20260620 19:10:01.689877 11452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:01.683665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.67:44381 (local address 127.0.0.1:58558)
0620 19:10:01.684081 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:01.684139 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:01.684299 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:01.685216 (+ 917us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:01.685222 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:01.685245 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:10:01.685422 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:01.685430 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:01.687060 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:01.687063 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:01.688469 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:01.688475 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:01.689411 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:01.689415 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:01.689496 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:01.689512 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 19:10:01.689556 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 19:10:01.689712 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"mutex_wait_us":133,"thread_start_us":109,"threads_started":1}
I20260620 19:10:01.690112 11455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:01.684395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58558 (local address 127.9.155.67:44381)
0620 19:10:01.684709 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:01.684720 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:01.684743 (+ 23us) server_negotiation.cc:407] Connection header received
0620 19:10:01.684816 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:01.684821 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:01.685053 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:01.685573 (+ 520us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:10:01.685716 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:01.686951 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:01.688571 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:01.689304 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:01.689679 (+ 375us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:01.689844 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:01.689892 (+ 48us) server_negotiation.cc:299] Negotiation successful
0620 19:10:01.689979 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":94,"threads_started":1}
I20260620 19:10:01.690366 11456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:01.684517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48172 (local address 127.9.155.66:35937)
0620 19:10:01.685294 (+ 777us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:01.685306 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:01.685326 (+ 20us) server_negotiation.cc:407] Connection header received
0620 19:10:01.685394 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:01.685400 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:01.685577 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:01.685706 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:10:01.685993 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:01.687078 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:01.688390 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:01.689217 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:01.689945 (+ 728us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:01.690088 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:01.690136 (+ 48us) server_negotiation.cc:299] Negotiation successful
0620 19:10:01.690220 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":683,"thread_start_us":89,"threads_started":1}
I20260620 19:10:01.690474 11453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:01.684003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.65:34885 (local address 127.0.0.1:37370)
0620 19:10:01.684969 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:01.684981 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:01.685055 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:01.685747 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:01.685749 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:01.685792 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:10:01.685899 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:01.685904 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:01.686876 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:01.686879 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:01.688463 (+ 1584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:01.688475 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:01.689285 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:01.689288 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:01.689397 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:01.689438 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 19:10:01.689561 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 19:10:01.690360 (+ 799us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":910,"mutex_wait_us":145,"spinlock_wait_cycles":13184,"thread_start_us":113,"threads_started":1}
I20260620 19:10:01.693122 11089 tablet_service.cc:1467] Processing CreateTablet for tablet a32db2f988f545a38e0a7a5c096ec087 (DEFAULT_TABLE table=test-table [id=5d8038d0d59b4ad7bbbf0a9285621366]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:10:01.693188 11355 tablet_service.cc:1467] Processing CreateTablet for tablet a32db2f988f545a38e0a7a5c096ec087 (DEFAULT_TABLE table=test-table [id=5d8038d0d59b4ad7bbbf0a9285621366]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:10:01.693188 11229 tablet_service.cc:1467] Processing CreateTablet for tablet a32db2f988f545a38e0a7a5c096ec087 (DEFAULT_TABLE table=test-table [id=5d8038d0d59b4ad7bbbf0a9285621366]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:10:01.694136 11089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a32db2f988f545a38e0a7a5c096ec087. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:01.694186 11229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a32db2f988f545a38e0a7a5c096ec087. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:01.694198 11355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a32db2f988f545a38e0a7a5c096ec087. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:01.701202 11458 tablet_bootstrap.cc:492] T a32db2f988f545a38e0a7a5c096ec087 P 3964b77bd96f435cab098d389d3b20f1: Bootstrap starting.
I20260620 19:10:01.702577 11460 tablet_bootstrap.cc:492] T a32db2f988f545a38e0a7a5c096ec087 P 0a8b3c70c2ec47589f5869aabbc0ce15: Bootstrap starting.
I20260620 19:10:01.703347 11458 tablet_bootstrap.cc:654] T a32db2f988f545a38e0a7a5c096ec087 P 3964b77bd96f435cab098d389d3b20f1: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:01.704058 11458 log.cc:824] T a32db2f988f545a38e0a7a5c096ec087 P 3964b77bd96f435cab098d389d3b20f1: Log is configured to *not* fsync() on all Append() calls
I20260620 19:10:01.704344 11460 tablet_bootstrap.cc:654] T a32db2f988f545a38e0a7a5c096ec087 P 0a8b3c70c2ec47589f5869aabbc0ce15: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:01.704600 11459 tablet_bootstrap.cc:492] T a32db2f988f545a38e0a7a5c096ec087 P 72317321300c441f8fb701add3d21f06: Bootstrap starting.
I20260620 19:10:01.705155 11460 log.cc:824] T a32db2f988f545a38e0a7a5c096ec087 P 0a8b3c70c2ec47589f5869aabbc0ce15: Log is configured to *not* fsync() on all Append() calls
I20260620 19:10:01.706063 11458 tablet_bootstrap.cc:492] T a32db2f988f545a38e0a7a5c096ec087 P 3964b77bd96f435cab098d389d3b20f1: No bootstrap required, opened a new log
I20260620 19:10:01.706276 11458 ts_tablet_manager.cc:1397] T a32db2f988f545a38e0a7a5c096ec087 P 3964b77bd96f435cab098d389d3b20f1: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 19:10:01.706743 11459 tablet_bootstrap.cc:654] T a32db2f988f545a38e0a7a5c096ec087 P 72317321300c441f8fb701add3d21f06: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:01.707433 11460 tablet_bootstrap.cc:492] T a32db2f988f545a38e0a7a5c096ec087 P 0a8b3c70c2ec47589f5869aabbc0ce15: No bootstrap required, opened a new log
I20260620 19:10:01.707568 11459 log.cc:824] T a32db2f988f545a38e0a7a5c096ec087 P 72317321300c441f8fb701add3d21f06: Log is configured to *not* fsync() on all Append() calls
I20260620 19:10:01.707626 11460 ts_tablet_manager.cc:1397] T a32db2f988f545a38e0a7a5c096ec087 P 0a8b3c70c2ec47589f5869aabbc0ce15: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260620 19:10:01.709645 11459 tablet_bootstrap.cc:492] T a32db2f988f545a38e0a7a5c096ec087 P 72317321300c441f8fb701add3d21f06: No bootstrap required, opened a new log
I20260620 19:10:01.709584 11458 raft_consensus.cc:348] T a32db2f988f545a38e0a7a5c096ec087 P 3964b77bd96f435cab098d389d3b20f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a8b3c70c2ec47589f5869aabbc0ce15" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 44381 } } peers { permanent_uuid: "72317321300c441f8fb701add3d21f06" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 35937 } } peers { permanent_uuid: "3964b77bd96f435cab098d389d3b20f1" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 34885 } }
I20260620 19:10:01.709893 11458 raft_consensus.cc:374] T a32db2f988f545a38e0a7a5c096ec087 P 3964b77bd96f435cab098d389d3b20f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:01.709851 11459 ts_tablet_manager.cc:1397] T a32db2f988f545a38e0a7a5c096ec087 P 72317321300c441f8fb701add3d21f06: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 19:10:01.709950 11458 raft_consensus.cc:729] T a32db2f988f545a38e0a7a5c096ec087 P 3964b77bd96f435cab098d389d3b20f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3964b77bd96f435cab098d389d3b20f1, State: Initialized, Role: FOLLOWER
I20260620 19:10:01.710493 11458 consensus_queue.cc:260] T a32db2f988f545a38e0a7a5c096ec087 P 3964b77bd96f435cab098d389d3b20f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a8b3c70c2ec47589f5869aabbc0ce15" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 44381 } } peers { permanent_uuid: "72317321300c441f8fb701add3d21f06" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 35937 } } peers { permanent_uuid: "3964b77bd96f435cab098d389d3b20f1" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 34885 } }
I20260620 19:10:01.711462 11458 ts_tablet_manager.cc:1428] T a32db2f988f545a38e0a7a5c096ec087 P 3964b77bd96f435cab098d389d3b20f1: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 19:10:01.711807 11155 heartbeater.cc:503] Master 127.9.155.126:43665 was elected leader, sending a full tablet report...
I20260620 19:10:01.711614 11460 raft_consensus.cc:348] T a32db2f988f545a38e0a7a5c096ec087 P 0a8b3c70c2ec47589f5869aabbc0ce15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a8b3c70c2ec47589f5869aabbc0ce15" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 44381 } } peers { permanent_uuid: "72317321300c441f8fb701add3d21f06" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 35937 } } peers { permanent_uuid: "3964b77bd96f435cab098d389d3b20f1" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 34885 } }
I20260620 19:10:01.711879 11460 raft_consensus.cc:374] T a32db2f988f545a38e0a7a5c096ec087 P 0a8b3c70c2ec47589f5869aabbc0ce15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:01.711944 11460 raft_consensus.cc:729] T a32db2f988f545a38e0a7a5c096ec087 P 0a8b3c70c2ec47589f5869aabbc0ce15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a8b3c70c2ec47589f5869aabbc0ce15, State: Initialized, Role: FOLLOWER
I20260620 19:10:01.712409 11460 consensus_queue.cc:260] T a32db2f988f545a38e0a7a5c096ec087 P 0a8b3c70c2ec47589f5869aabbc0ce15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a8b3c70c2ec47589f5869aabbc0ce15" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 44381 } } peers { permanent_uuid: "72317321300c441f8fb701add3d21f06" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 35937 } } peers { permanent_uuid: "3964b77bd96f435cab098d389d3b20f1" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 34885 } }
I20260620 19:10:01.713083 11435 heartbeater.cc:503] Master 127.9.155.126:43665 was elected leader, sending a full tablet report...
I20260620 19:10:01.713807 11459 raft_consensus.cc:348] T a32db2f988f545a38e0a7a5c096ec087 P 72317321300c441f8fb701add3d21f06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a8b3c70c2ec47589f5869aabbc0ce15" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 44381 } } peers { permanent_uuid: "72317321300c441f8fb701add3d21f06" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 35937 } } peers { permanent_uuid: "3964b77bd96f435cab098d389d3b20f1" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 34885 } }
I20260620 19:10:01.714069 11459 raft_consensus.cc:374] T a32db2f988f545a38e0a7a5c096ec087 P 72317321300c441f8fb701add3d21f06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:01.714113 11459 raft_consensus.cc:729] T a32db2f988f545a38e0a7a5c096ec087 P 72317321300c441f8fb701add3d21f06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72317321300c441f8fb701add3d21f06, State: Initialized, Role: FOLLOWER
I20260620 19:10:01.714483 11460 ts_tablet_manager.cc:1428] T a32db2f988f545a38e0a7a5c096ec087 P 0a8b3c70c2ec47589f5869aabbc0ce15: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260620 19:10:01.714582 11459 consensus_queue.cc:260] T a32db2f988f545a38e0a7a5c096ec087 P 72317321300c441f8fb701add3d21f06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a8b3c70c2ec47589f5869aabbc0ce15" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 44381 } } peers { permanent_uuid: "72317321300c441f8fb701add3d21f06" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 35937 } } peers { permanent_uuid: "3964b77bd96f435cab098d389d3b20f1" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 34885 } }
I20260620 19:10:01.715178 11295 heartbeater.cc:503] Master 127.9.155.126:43665 was elected leader, sending a full tablet report...
I20260620 19:10:01.715538 11459 ts_tablet_manager.cc:1428] T a32db2f988f545a38e0a7a5c096ec087 P 72317321300c441f8fb701add3d21f06: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260620 19:10:01.875970 11436 tablet.cc:2369] T a32db2f988f545a38e0a7a5c096ec087 P 0a8b3c70c2ec47589f5869aabbc0ce15: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:10:01.896090 11464 raft_consensus.cc:482] T a32db2f988f545a38e0a7a5c096ec087 P 3964b77bd96f435cab098d389d3b20f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:10:01.896410 11464 raft_consensus.cc:504] T a32db2f988f545a38e0a7a5c096ec087 P 3964b77bd96f435cab098d389d3b20f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a8b3c70c2ec47589f5869aabbc0ce15" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 44381 } } peers { permanent_uuid: "72317321300c441f8fb701add3d21f06" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 35937 } } peers { permanent_uuid: "3964b77bd96f435cab098d389d3b20f1" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 34885 } }
I20260620 19:10:01.897615 11464 leader_election.cc:288] T a32db2f988f545a38e0a7a5c096ec087 P 3964b77bd96f435cab098d389d3b20f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a8b3c70c2ec47589f5869aabbc0ce15 (127.9.155.67:44381), 72317321300c441f8fb701add3d21f06 (127.9.155.66:35937)
I20260620 19:10:01.901445 11467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:01.897847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.67:44381 (local address 127.9.155.65:46965)
0620 19:10:01.898157 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:01.898171 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:01.898296 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:01.898575 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:01.898578 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:01.898596 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:10:01.898706 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:01.898720 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:01.899537 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:01.899540 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:01.900409 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:01.900419 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:01.901025 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:01.901029 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:01.901122 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:01.901145 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 19:10:01.901210 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 19:10:01.901282 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"mutex_wait_us":18,"thread_start_us":73,"threads_started":1}
I20260620 19:10:01.901427 11455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:01.897970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.65:46965 (local address 127.9.155.67:44381)
0620 19:10:01.898093 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:01.898098 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:01.898250 (+ 152us) server_negotiation.cc:407] Connection header received
0620 19:10:01.898431 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:01.898435 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:01.898483 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:01.898556 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:10:01.898826 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:01.899414 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:01.900532 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:01.900917 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:01.901025 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:01.901090 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:01.901243 (+ 153us) server_negotiation.cc:299] Negotiation successful
0620 19:10:01.901317 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 19:10:01.901595 11456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:01.898084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.65:46559 (local address 127.9.155.66:35937)
0620 19:10:01.898262 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:01.898267 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:01.898547 (+ 280us) server_negotiation.cc:407] Connection header received
0620 19:10:01.898661 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:01.898664 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:01.898704 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:01.898764 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:10:01.899006 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:01.899607 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:01.900679 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:01.901095 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:01.901202 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:01.901251 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:01.901419 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 19:10:01.901491 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 19:10:01.901610 11468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:01.897990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.66:35937 (local address 127.9.155.65:46559)
0620 19:10:01.898467 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:01.898478 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:01.898567 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:01.898780 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:01.898783 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:01.898798 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:10:01.898916 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:01.898921 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:01.899705 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:01.899708 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:01.900572 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:01.900581 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:01.901205 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:01.901209 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:01.901318 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:01.901334 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 19:10:01.901397 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 19:10:01.901463 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"mutex_wait_us":18,"thread_start_us":129,"threads_started":1}
I20260620 19:10:01.902326 11389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a32db2f988f545a38e0a7a5c096ec087" candidate_uuid: "3964b77bd96f435cab098d389d3b20f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a8b3c70c2ec47589f5869aabbc0ce15" is_pre_election: true
I20260620 19:10:01.902532 11249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a32db2f988f545a38e0a7a5c096ec087" candidate_uuid: "3964b77bd96f435cab098d389d3b20f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72317321300c441f8fb701add3d21f06" is_pre_election: true
I20260620 19:10:01.902642 11389 raft_consensus.cc:2454] T a32db2f988f545a38e0a7a5c096ec087 P 0a8b3c70c2ec47589f5869aabbc0ce15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3964b77bd96f435cab098d389d3b20f1 in term 0.
I20260620 19:10:01.902832 11249 raft_consensus.cc:2454] T a32db2f988f545a38e0a7a5c096ec087 P 72317321300c441f8fb701add3d21f06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3964b77bd96f435cab098d389d3b20f1 in term 0.
I20260620 19:10:01.903156 11042 leader_election.cc:302] T a32db2f988f545a38e0a7a5c096ec087 P 3964b77bd96f435cab098d389d3b20f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a8b3c70c2ec47589f5869aabbc0ce15, 3964b77bd96f435cab098d389d3b20f1; no voters:
I20260620 19:10:01.903434 11464 raft_consensus.cc:2781] T a32db2f988f545a38e0a7a5c096ec087 P 3964b77bd96f435cab098d389d3b20f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:10:01.903517 11464 raft_consensus.cc:482] T a32db2f988f545a38e0a7a5c096ec087 P 3964b77bd96f435cab098d389d3b20f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:10:01.903564 11464 raft_consensus.cc:3037] T a32db2f988f545a38e0a7a5c096ec087 P 3964b77bd96f435cab098d389d3b20f1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:01.904645 11464 raft_consensus.cc:504] T a32db2f988f545a38e0a7a5c096ec087 P 3964b77bd96f435cab098d389d3b20f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a8b3c70c2ec47589f5869aabbc0ce15" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 44381 } } peers { permanent_uuid: "72317321300c441f8fb701add3d21f06" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 35937 } } peers { permanent_uuid: "3964b77bd96f435cab098d389d3b20f1" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 34885 } }
I20260620 19:10:01.904973 11464 leader_election.cc:288] T a32db2f988f545a38e0a7a5c096ec087 P 3964b77bd96f435cab098d389d3b20f1 [CANDIDATE]: Term 1 election: Requested vote from peers 0a8b3c70c2ec47589f5869aabbc0ce15 (127.9.155.67:44381), 72317321300c441f8fb701add3d21f06 (127.9.155.66:35937)
I20260620 19:10:01.905320 11389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a32db2f988f545a38e0a7a5c096ec087" candidate_uuid: "3964b77bd96f435cab098d389d3b20f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a8b3c70c2ec47589f5869aabbc0ce15"
I20260620 19:10:01.905339 11249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a32db2f988f545a38e0a7a5c096ec087" candidate_uuid: "3964b77bd96f435cab098d389d3b20f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72317321300c441f8fb701add3d21f06"
I20260620 19:10:01.905429 11389 raft_consensus.cc:3037] T a32db2f988f545a38e0a7a5c096ec087 P 0a8b3c70c2ec47589f5869aabbc0ce15 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:01.905452 11249 raft_consensus.cc:3037] T a32db2f988f545a38e0a7a5c096ec087 P 72317321300c441f8fb701add3d21f06 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:01.906504 11249 raft_consensus.cc:2454] T a32db2f988f545a38e0a7a5c096ec087 P 72317321300c441f8fb701add3d21f06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3964b77bd96f435cab098d389d3b20f1 in term 1.
I20260620 19:10:01.906780 11389 raft_consensus.cc:2454] T a32db2f988f545a38e0a7a5c096ec087 P 0a8b3c70c2ec47589f5869aabbc0ce15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3964b77bd96f435cab098d389d3b20f1 in term 1.
I20260620 19:10:01.906802 11043 leader_election.cc:302] T a32db2f988f545a38e0a7a5c096ec087 P 3964b77bd96f435cab098d389d3b20f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3964b77bd96f435cab098d389d3b20f1, 72317321300c441f8fb701add3d21f06; no voters:
I20260620 19:10:01.906970 11464 raft_consensus.cc:2781] T a32db2f988f545a38e0a7a5c096ec087 P 3964b77bd96f435cab098d389d3b20f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:10:01.907164 11464 raft_consensus.cc:686] T a32db2f988f545a38e0a7a5c096ec087 P 3964b77bd96f435cab098d389d3b20f1 [term 1 LEADER]: Becoming Leader. State: Replica: 3964b77bd96f435cab098d389d3b20f1, State: Running, Role: LEADER
I20260620 19:10:01.907505 11464 consensus_queue.cc:237] T a32db2f988f545a38e0a7a5c096ec087 P 3964b77bd96f435cab098d389d3b20f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a8b3c70c2ec47589f5869aabbc0ce15" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 44381 } } peers { permanent_uuid: "72317321300c441f8fb701add3d21f06" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 35937 } } peers { permanent_uuid: "3964b77bd96f435cab098d389d3b20f1" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 34885 } }
I20260620 19:10:01.910730 10949 catalog_manager.cc:5495] T a32db2f988f545a38e0a7a5c096ec087 P 3964b77bd96f435cab098d389d3b20f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3964b77bd96f435cab098d389d3b20f1 (127.9.155.65). New cstate: current_term: 1 leader_uuid: "3964b77bd96f435cab098d389d3b20f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a8b3c70c2ec47589f5869aabbc0ce15" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 44381 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72317321300c441f8fb701add3d21f06" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 35937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3964b77bd96f435cab098d389d3b20f1" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 34885 } health_report { overall_health: HEALTHY } } }
W20260620 19:10:01.911114 11296 tablet.cc:2369] T a32db2f988f545a38e0a7a5c096ec087 P 72317321300c441f8fb701add3d21f06: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:10:01.933596 11457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:01.929732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37374 (local address 127.9.155.65:34885)
0620 19:10:01.929869 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:01.929873 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:01.929888 (+ 15us) server_negotiation.cc:407] Connection header received
0620 19:10:01.930100 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:01.930102 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:01.930172 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:01.930266 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:10:01.930548 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:01.931109 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:01.932052 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:01.932313 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:01.932406 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:01.932762 (+ 356us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:10:01.933188 (+ 426us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:10:01.933294 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:01.933388 (+ 94us) server_negotiation.cc:299] Negotiation successful
0620 19:10:01.933451 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 19:10:01.939349 11389 raft_consensus.cc:1261] T a32db2f988f545a38e0a7a5c096ec087 P 0a8b3c70c2ec47589f5869aabbc0ce15 [term 1 FOLLOWER]: Refusing update from remote peer 3964b77bd96f435cab098d389d3b20f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:10:01.939404 11249 raft_consensus.cc:1261] T a32db2f988f545a38e0a7a5c096ec087 P 72317321300c441f8fb701add3d21f06 [term 1 FOLLOWER]: Refusing update from remote peer 3964b77bd96f435cab098d389d3b20f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:10:01.940053 11469 consensus_queue.cc:1035] T a32db2f988f545a38e0a7a5c096ec087 P 3964b77bd96f435cab098d389d3b20f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a8b3c70c2ec47589f5869aabbc0ce15" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 44381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:10:01.940249 11464 consensus_queue.cc:1035] T a32db2f988f545a38e0a7a5c096ec087 P 3964b77bd96f435cab098d389d3b20f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72317321300c441f8fb701add3d21f06" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 35937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 19:10:01.945462 11156 tablet.cc:2369] T a32db2f988f545a38e0a7a5c096ec087 P 3964b77bd96f435cab098d389d3b20f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:10:01.947808 11474 mvcc.cc:205] Tried to move back new op lower bound from 7299000737536524288 to 7299000737415725056. Current Snapshot: MvccSnapshot[applied={T|T < 7299000737536524288}]
I20260620 19:10:01.948117 11473 mvcc.cc:205] Tried to move back new op lower bound from 7299000737536524288 to 7299000737415725056. Current Snapshot: MvccSnapshot[applied={T|T < 7299000737536524288}]
I20260620 19:10:01.958812 10952 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50826:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:10:01.958997 10952 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50826:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:10:01.961608 10952 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 05b0609251bd43e5aeb2a0b1a38f4f49: Sending DeleteTablet for 3 replicas of tablet a32db2f988f545a38e0a7a5c096ec087
I20260620 19:10:01.962466 11355 tablet_service.cc:1514] Processing DeleteTablet for tablet a32db2f988f545a38e0a7a5c096ec087 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:10:01 UTC) from {username='kudu', principal='kudu/127.9.155.126@KRBTEST.COM'} at 127.0.0.1:58558
I20260620 19:10:01.962603 11229 tablet_service.cc:1514] Processing DeleteTablet for tablet a32db2f988f545a38e0a7a5c096ec087 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:10:01 UTC) from {username='kudu', principal='kudu/127.9.155.126@KRBTEST.COM'} at 127.0.0.1:48172
I20260620 19:10:01.963006 11482 tablet_replica.cc:330] T a32db2f988f545a38e0a7a5c096ec087 P 72317321300c441f8fb701add3d21f06: stopping tablet replica
I20260620 19:10:01.963028 11089 tablet_service.cc:1514] Processing DeleteTablet for tablet a32db2f988f545a38e0a7a5c096ec087 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:10:01 UTC) from {username='kudu', principal='kudu/127.9.155.126@KRBTEST.COM'} at 127.0.0.1:37370
I20260620 19:10:01.963578 11483 tablet_replica.cc:330] T a32db2f988f545a38e0a7a5c096ec087 P 3964b77bd96f435cab098d389d3b20f1: stopping tablet replica
I20260620 19:10:01.963578 11482 raft_consensus.cc:2229] T a32db2f988f545a38e0a7a5c096ec087 P 72317321300c441f8fb701add3d21f06 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:10:01.963874 11483 raft_consensus.cc:2229] T a32db2f988f545a38e0a7a5c096ec087 P 3964b77bd96f435cab098d389d3b20f1 [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:10:01.964076 11482 raft_consensus.cc:2258] T a32db2f988f545a38e0a7a5c096ec087 P 72317321300c441f8fb701add3d21f06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:10:01.964089 10952 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50826:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 19:10:01.964318 11483 raft_consensus.cc:2258] T a32db2f988f545a38e0a7a5c096ec087 P 3964b77bd96f435cab098d389d3b20f1 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 19:10:01.964581 10952 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:10:01.964775 11481 tablet_replica.cc:330] T a32db2f988f545a38e0a7a5c096ec087 P 0a8b3c70c2ec47589f5869aabbc0ce15: stopping tablet replica
I20260620 19:10:01.965076 11481 raft_consensus.cc:2229] T a32db2f988f545a38e0a7a5c096ec087 P 0a8b3c70c2ec47589f5869aabbc0ce15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:10:01.965390 11481 raft_consensus.cc:2258] T a32db2f988f545a38e0a7a5c096ec087 P 0a8b3c70c2ec47589f5869aabbc0ce15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:10:01.966028 11483 ts_tablet_manager.cc:1905] T a32db2f988f545a38e0a7a5c096ec087 P 3964b77bd96f435cab098d389d3b20f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:10:01.966028 11482 ts_tablet_manager.cc:1905] T a32db2f988f545a38e0a7a5c096ec087 P 72317321300c441f8fb701add3d21f06: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:10:01.969239 11482 ts_tablet_manager.cc:1918] T a32db2f988f545a38e0a7a5c096ec087 P 72317321300c441f8fb701add3d21f06: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:10:01.969239 11483 ts_tablet_manager.cc:1918] T a32db2f988f545a38e0a7a5c096ec087 P 3964b77bd96f435cab098d389d3b20f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:10:01.969362 11483 log.cc:1196] T a32db2f988f545a38e0a7a5c096ec087 P 3964b77bd96f435cab098d389d3b20f1: Deleting WAL directory at /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/minicluster-data/ts-0/wal/wals/a32db2f988f545a38e0a7a5c096ec087
I20260620 19:10:01.969507 11482 log.cc:1196] T a32db2f988f545a38e0a7a5c096ec087 P 72317321300c441f8fb701add3d21f06: Deleting WAL directory at /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/minicluster-data/ts-1/wal/wals/a32db2f988f545a38e0a7a5c096ec087
I20260620 19:10:01.969699 11483 ts_tablet_manager.cc:1939] T a32db2f988f545a38e0a7a5c096ec087 P 3964b77bd96f435cab098d389d3b20f1: Deleting consensus metadata
I20260620 19:10:01.969967 11482 ts_tablet_manager.cc:1939] T a32db2f988f545a38e0a7a5c096ec087 P 72317321300c441f8fb701add3d21f06: Deleting consensus metadata
I20260620 19:10:01.971107 11229 tablet_service.cc:1467] Processing CreateTablet for tablet fb9d48d591fb45308d690147e79944e7 (DEFAULT_TABLE table=test-table [id=ca9eed32276e4845b7446e76421cce4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:10:01.971408 11229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb9d48d591fb45308d690147e79944e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:01.971554 10937 catalog_manager.cc:4841] TS 3964b77bd96f435cab098d389d3b20f1 (127.9.155.65:34885): tablet a32db2f988f545a38e0a7a5c096ec087 (table test-table [id=5d8038d0d59b4ad7bbbf0a9285621366]) successfully deleted
I20260620 19:10:01.972121 11089 tablet_service.cc:1467] Processing CreateTablet for tablet fb9d48d591fb45308d690147e79944e7 (DEFAULT_TABLE table=test-table [id=ca9eed32276e4845b7446e76421cce4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:10:01.972395 11089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb9d48d591fb45308d690147e79944e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:01.972434 10939 catalog_manager.cc:4841] TS 72317321300c441f8fb701add3d21f06 (127.9.155.66:35937): tablet a32db2f988f545a38e0a7a5c096ec087 (table test-table [id=5d8038d0d59b4ad7bbbf0a9285621366]) successfully deleted
I20260620 19:10:01.973045 11355 tablet_service.cc:1467] Processing CreateTablet for tablet fb9d48d591fb45308d690147e79944e7 (DEFAULT_TABLE table=test-table [id=ca9eed32276e4845b7446e76421cce4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:10:01.973388 11355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb9d48d591fb45308d690147e79944e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:01.974452 11459 tablet_bootstrap.cc:492] T fb9d48d591fb45308d690147e79944e7 P 72317321300c441f8fb701add3d21f06: Bootstrap starting.
I20260620 19:10:01.974717 11458 tablet_bootstrap.cc:492] T fb9d48d591fb45308d690147e79944e7 P 3964b77bd96f435cab098d389d3b20f1: Bootstrap starting.
I20260620 19:10:01.975386 11459 tablet_bootstrap.cc:654] T fb9d48d591fb45308d690147e79944e7 P 72317321300c441f8fb701add3d21f06: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:01.975680 11458 tablet_bootstrap.cc:654] T fb9d48d591fb45308d690147e79944e7 P 3964b77bd96f435cab098d389d3b20f1: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:01.975795 11460 tablet_bootstrap.cc:492] T fb9d48d591fb45308d690147e79944e7 P 0a8b3c70c2ec47589f5869aabbc0ce15: Bootstrap starting.
I20260620 19:10:01.976778 11460 tablet_bootstrap.cc:654] T fb9d48d591fb45308d690147e79944e7 P 0a8b3c70c2ec47589f5869aabbc0ce15: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:01.977180 11459 tablet_bootstrap.cc:492] T fb9d48d591fb45308d690147e79944e7 P 72317321300c441f8fb701add3d21f06: No bootstrap required, opened a new log
I20260620 19:10:01.977272 11459 ts_tablet_manager.cc:1397] T fb9d48d591fb45308d690147e79944e7 P 72317321300c441f8fb701add3d21f06: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 19:10:01.977732 11459 raft_consensus.cc:348] T fb9d48d591fb45308d690147e79944e7 P 72317321300c441f8fb701add3d21f06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3964b77bd96f435cab098d389d3b20f1" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 34885 } } peers { permanent_uuid: "72317321300c441f8fb701add3d21f06" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 35937 } } peers { permanent_uuid: "0a8b3c70c2ec47589f5869aabbc0ce15" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 44381 } }
I20260620 19:10:01.977861 11459 raft_consensus.cc:374] T fb9d48d591fb45308d690147e79944e7 P 72317321300c441f8fb701add3d21f06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:01.977891 11459 raft_consensus.cc:729] T fb9d48d591fb45308d690147e79944e7 P 72317321300c441f8fb701add3d21f06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72317321300c441f8fb701add3d21f06, State: Initialized, Role: FOLLOWER
I20260620 19:10:01.978001 11481 ts_tablet_manager.cc:1905] T a32db2f988f545a38e0a7a5c096ec087 P 0a8b3c70c2ec47589f5869aabbc0ce15: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:10:01.978662 11459 consensus_queue.cc:260] T fb9d48d591fb45308d690147e79944e7 P 72317321300c441f8fb701add3d21f06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3964b77bd96f435cab098d389d3b20f1" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 34885 } } peers { permanent_uuid: "72317321300c441f8fb701add3d21f06" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 35937 } } peers { permanent_uuid: "0a8b3c70c2ec47589f5869aabbc0ce15" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 44381 } }
I20260620 19:10:01.979053 11459 ts_tablet_manager.cc:1428] T fb9d48d591fb45308d690147e79944e7 P 72317321300c441f8fb701add3d21f06: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:10:01.979915 11458 tablet_bootstrap.cc:492] T fb9d48d591fb45308d690147e79944e7 P 3964b77bd96f435cab098d389d3b20f1: No bootstrap required, opened a new log
I20260620 19:10:01.980017 11458 ts_tablet_manager.cc:1397] T fb9d48d591fb45308d690147e79944e7 P 3964b77bd96f435cab098d389d3b20f1: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260620 19:10:01.980727 11481 ts_tablet_manager.cc:1918] T a32db2f988f545a38e0a7a5c096ec087 P 0a8b3c70c2ec47589f5869aabbc0ce15: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:10:01.980836 11481 log.cc:1196] T a32db2f988f545a38e0a7a5c096ec087 P 0a8b3c70c2ec47589f5869aabbc0ce15: Deleting WAL directory at /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/minicluster-data/ts-2/wal/wals/a32db2f988f545a38e0a7a5c096ec087
I20260620 19:10:01.981150 11481 ts_tablet_manager.cc:1939] T a32db2f988f545a38e0a7a5c096ec087 P 0a8b3c70c2ec47589f5869aabbc0ce15: Deleting consensus metadata
I20260620 19:10:01.981710 11458 raft_consensus.cc:348] T fb9d48d591fb45308d690147e79944e7 P 3964b77bd96f435cab098d389d3b20f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3964b77bd96f435cab098d389d3b20f1" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 34885 } } peers { permanent_uuid: "72317321300c441f8fb701add3d21f06" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 35937 } } peers { permanent_uuid: "0a8b3c70c2ec47589f5869aabbc0ce15" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 44381 } }
I20260620 19:10:01.981843 11458 raft_consensus.cc:374] T fb9d48d591fb45308d690147e79944e7 P 3964b77bd96f435cab098d389d3b20f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:01.981894 11458 raft_consensus.cc:729] T fb9d48d591fb45308d690147e79944e7 P 3964b77bd96f435cab098d389d3b20f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3964b77bd96f435cab098d389d3b20f1, State: Initialized, Role: FOLLOWER
I20260620 19:10:01.982023 11458 consensus_queue.cc:260] T fb9d48d591fb45308d690147e79944e7 P 3964b77bd96f435cab098d389d3b20f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3964b77bd96f435cab098d389d3b20f1" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 34885 } } peers { permanent_uuid: "72317321300c441f8fb701add3d21f06" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 35937 } } peers { permanent_uuid: "0a8b3c70c2ec47589f5869aabbc0ce15" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 44381 } }
I20260620 19:10:01.982141 11460 tablet_bootstrap.cc:492] T fb9d48d591fb45308d690147e79944e7 P 0a8b3c70c2ec47589f5869aabbc0ce15: No bootstrap required, opened a new log
I20260620 19:10:01.982221 11460 ts_tablet_manager.cc:1397] T fb9d48d591fb45308d690147e79944e7 P 0a8b3c70c2ec47589f5869aabbc0ce15: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260620 19:10:01.982288 11458 ts_tablet_manager.cc:1428] T fb9d48d591fb45308d690147e79944e7 P 3964b77bd96f435cab098d389d3b20f1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:10:01.982659 11460 raft_consensus.cc:348] T fb9d48d591fb45308d690147e79944e7 P 0a8b3c70c2ec47589f5869aabbc0ce15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3964b77bd96f435cab098d389d3b20f1" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 34885 } } peers { permanent_uuid: "72317321300c441f8fb701add3d21f06" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 35937 } } peers { permanent_uuid: "0a8b3c70c2ec47589f5869aabbc0ce15" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 44381 } }
I20260620 19:10:01.982769 11460 raft_consensus.cc:374] T fb9d48d591fb45308d690147e79944e7 P 0a8b3c70c2ec47589f5869aabbc0ce15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:01.982800 11460 raft_consensus.cc:729] T fb9d48d591fb45308d690147e79944e7 P 0a8b3c70c2ec47589f5869aabbc0ce15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a8b3c70c2ec47589f5869aabbc0ce15, State: Initialized, Role: FOLLOWER
I20260620 19:10:01.982932 11460 consensus_queue.cc:260] T fb9d48d591fb45308d690147e79944e7 P 0a8b3c70c2ec47589f5869aabbc0ce15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3964b77bd96f435cab098d389d3b20f1" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 34885 } } peers { permanent_uuid: "72317321300c441f8fb701add3d21f06" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 35937 } } peers { permanent_uuid: "0a8b3c70c2ec47589f5869aabbc0ce15" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 44381 } }
I20260620 19:10:01.983196 11460 ts_tablet_manager.cc:1428] T fb9d48d591fb45308d690147e79944e7 P 0a8b3c70c2ec47589f5869aabbc0ce15: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:10:01.983477 10938 catalog_manager.cc:4841] TS 0a8b3c70c2ec47589f5869aabbc0ce15 (127.9.155.67:44381): tablet a32db2f988f545a38e0a7a5c096ec087 (table test-table [id=5d8038d0d59b4ad7bbbf0a9285621366]) successfully deleted
I20260620 19:10:02.051005 11465 raft_consensus.cc:482] T fb9d48d591fb45308d690147e79944e7 P 0a8b3c70c2ec47589f5869aabbc0ce15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:10:02.051156 11465 raft_consensus.cc:504] T fb9d48d591fb45308d690147e79944e7 P 0a8b3c70c2ec47589f5869aabbc0ce15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3964b77bd96f435cab098d389d3b20f1" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 34885 } } peers { permanent_uuid: "72317321300c441f8fb701add3d21f06" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 35937 } } peers { permanent_uuid: "0a8b3c70c2ec47589f5869aabbc0ce15" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 44381 } }
I20260620 19:10:02.052321 11465 leader_election.cc:288] T fb9d48d591fb45308d690147e79944e7 P 0a8b3c70c2ec47589f5869aabbc0ce15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3964b77bd96f435cab098d389d3b20f1 (127.9.155.65:34885), 72317321300c441f8fb701add3d21f06 (127.9.155.66:35937)
I20260620 19:10:02.057096 11328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:02.052504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.65:34885 (local address 127.9.155.67:56047)
0620 19:10:02.052910 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:02.052926 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:02.053035 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:02.053271 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:02.053274 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:02.053288 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:10:02.053415 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:02.053423 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:02.054434 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:02.054438 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:02.055608 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:02.055621 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:02.056591 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:02.056595 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:02.056729 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:02.056755 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 19:10:02.056840 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 19:10:02.056933 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"mutex_wait_us":23}
I20260620 19:10:02.057158 11456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:02.052771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.67:38799 (local address 127.9.155.66:35937)
0620 19:10:02.052894 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:02.052898 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:02.053027 (+ 129us) server_negotiation.cc:407] Connection header received
0620 19:10:02.053211 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:02.053214 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:02.053263 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:02.053328 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:10:02.053632 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:02.054145 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:02.055749 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:02.056422 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:02.056541 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:02.056590 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:02.056935 (+ 345us) server_negotiation.cc:299] Negotiation successful
0620 19:10:02.057026 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 19:10:02.057533 11487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:02.052586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.66:35937 (local address 127.9.155.67:38799)
0620 19:10:02.052941 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:02.052954 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:02.053097 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:02.053378 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:02.053381 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:02.053394 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:10:02.053498 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:02.053504 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:02.054246 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:02.054249 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:02.055611 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:02.055621 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:02.056663 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:02.056667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:02.056783 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:02.056805 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 19:10:02.057299 (+ 494us) client_negotiation.cc:240] Negotiation successful
0620 19:10:02.057382 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":25,"thread_start_us":102,"threads_started":1}
I20260620 19:10:02.057749 11457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:02.052603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.67:56047 (local address 127.9.155.65:34885)
0620 19:10:02.052721 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:02.052725 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:02.053007 (+ 282us) server_negotiation.cc:407] Connection header received
0620 19:10:02.053137 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:02.053140 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:02.053192 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:02.053257 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:10:02.053539 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:02.054315 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:02.055774 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:02.056237 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:02.056359 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:02.056506 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:02.057540 (+ 1034us) server_negotiation.cc:299] Negotiation successful
0620 19:10:02.057631 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 19:10:02.058064 11249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb9d48d591fb45308d690147e79944e7" candidate_uuid: "0a8b3c70c2ec47589f5869aabbc0ce15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72317321300c441f8fb701add3d21f06" is_pre_election: true
I20260620 19:10:02.058238 11249 raft_consensus.cc:2454] T fb9d48d591fb45308d690147e79944e7 P 72317321300c441f8fb701add3d21f06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a8b3c70c2ec47589f5869aabbc0ce15 in term 0.
I20260620 19:10:02.058316 11109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb9d48d591fb45308d690147e79944e7" candidate_uuid: "0a8b3c70c2ec47589f5869aabbc0ce15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3964b77bd96f435cab098d389d3b20f1" is_pre_election: true
I20260620 19:10:02.058574 11109 raft_consensus.cc:2454] T fb9d48d591fb45308d690147e79944e7 P 3964b77bd96f435cab098d389d3b20f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a8b3c70c2ec47589f5869aabbc0ce15 in term 0.
I20260620 19:10:02.058617 11323 leader_election.cc:302] T fb9d48d591fb45308d690147e79944e7 P 0a8b3c70c2ec47589f5869aabbc0ce15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a8b3c70c2ec47589f5869aabbc0ce15, 72317321300c441f8fb701add3d21f06; no voters:
I20260620 19:10:02.058889 11465 raft_consensus.cc:2781] T fb9d48d591fb45308d690147e79944e7 P 0a8b3c70c2ec47589f5869aabbc0ce15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:10:02.058954 11465 raft_consensus.cc:482] T fb9d48d591fb45308d690147e79944e7 P 0a8b3c70c2ec47589f5869aabbc0ce15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:10:02.058996 11465 raft_consensus.cc:3037] T fb9d48d591fb45308d690147e79944e7 P 0a8b3c70c2ec47589f5869aabbc0ce15 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:02.060439 11465 raft_consensus.cc:504] T fb9d48d591fb45308d690147e79944e7 P 0a8b3c70c2ec47589f5869aabbc0ce15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3964b77bd96f435cab098d389d3b20f1" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 34885 } } peers { permanent_uuid: "72317321300c441f8fb701add3d21f06" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 35937 } } peers { permanent_uuid: "0a8b3c70c2ec47589f5869aabbc0ce15" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 44381 } }
I20260620 19:10:02.060887 11465 leader_election.cc:288] T fb9d48d591fb45308d690147e79944e7 P 0a8b3c70c2ec47589f5869aabbc0ce15 [CANDIDATE]: Term 1 election: Requested vote from peers 3964b77bd96f435cab098d389d3b20f1 (127.9.155.65:34885), 72317321300c441f8fb701add3d21f06 (127.9.155.66:35937)
I20260620 19:10:02.061439 11249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb9d48d591fb45308d690147e79944e7" candidate_uuid: "0a8b3c70c2ec47589f5869aabbc0ce15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72317321300c441f8fb701add3d21f06"
I20260620 19:10:02.061460 11109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb9d48d591fb45308d690147e79944e7" candidate_uuid: "0a8b3c70c2ec47589f5869aabbc0ce15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3964b77bd96f435cab098d389d3b20f1"
I20260620 19:10:02.061578 11249 raft_consensus.cc:3037] T fb9d48d591fb45308d690147e79944e7 P 72317321300c441f8fb701add3d21f06 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:02.061581 11109 raft_consensus.cc:3037] T fb9d48d591fb45308d690147e79944e7 P 3964b77bd96f435cab098d389d3b20f1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:02.062671 11109 raft_consensus.cc:2454] T fb9d48d591fb45308d690147e79944e7 P 3964b77bd96f435cab098d389d3b20f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a8b3c70c2ec47589f5869aabbc0ce15 in term 1.
I20260620 19:10:02.062810 11249 raft_consensus.cc:2454] T fb9d48d591fb45308d690147e79944e7 P 72317321300c441f8fb701add3d21f06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a8b3c70c2ec47589f5869aabbc0ce15 in term 1.
I20260620 19:10:02.062983 11321 leader_election.cc:302] T fb9d48d591fb45308d690147e79944e7 P 0a8b3c70c2ec47589f5869aabbc0ce15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a8b3c70c2ec47589f5869aabbc0ce15, 3964b77bd96f435cab098d389d3b20f1; no voters:
I20260620 19:10:02.063179 11465 raft_consensus.cc:2781] T fb9d48d591fb45308d690147e79944e7 P 0a8b3c70c2ec47589f5869aabbc0ce15 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:10:02.063392 11465 raft_consensus.cc:686] T fb9d48d591fb45308d690147e79944e7 P 0a8b3c70c2ec47589f5869aabbc0ce15 [term 1 LEADER]: Becoming Leader. State: Replica: 0a8b3c70c2ec47589f5869aabbc0ce15, State: Running, Role: LEADER
I20260620 19:10:02.063664 11465 consensus_queue.cc:237] T fb9d48d591fb45308d690147e79944e7 P 0a8b3c70c2ec47589f5869aabbc0ce15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3964b77bd96f435cab098d389d3b20f1" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 34885 } } peers { permanent_uuid: "72317321300c441f8fb701add3d21f06" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 35937 } } peers { permanent_uuid: "0a8b3c70c2ec47589f5869aabbc0ce15" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 44381 } }
I20260620 19:10:02.066272 10952 catalog_manager.cc:5495] T fb9d48d591fb45308d690147e79944e7 P 0a8b3c70c2ec47589f5869aabbc0ce15 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a8b3c70c2ec47589f5869aabbc0ce15 (127.9.155.67). New cstate: current_term: 1 leader_uuid: "0a8b3c70c2ec47589f5869aabbc0ce15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3964b77bd96f435cab098d389d3b20f1" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 34885 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72317321300c441f8fb701add3d21f06" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 35937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a8b3c70c2ec47589f5869aabbc0ce15" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 44381 } health_report { overall_health: HEALTHY } } }
Jun 20 19:10:02 dist-test-slave-n3fx krb5kdc[10907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982600, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.126@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:10:02.115113 11001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:02.106117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50830 (local address 127.9.155.126:43665)
0620 19:10:02.106258 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:02.106263 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:02.106450 (+ 187us) server_negotiation.cc:407] Connection header received
0620 19:10:02.106592 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:02.106594 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:02.106630 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:02.106690 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:02.107376 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:02.107904 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:02.108744 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:02.108951 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:02.109038 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:02.111860 (+ 2822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:02.111863 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:02.111865 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:02.111893 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:02.113400 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:02.113906 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:02.113910 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:02.113911 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:02.113984 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:02.114217 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:02.114220 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:02.114222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:02.114412 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:02.114496 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:02.114800 (+ 304us) server_negotiation.cc:299] Negotiation successful
0620 19:10:02.114983 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":78,"server-negotiator.queue_time_us":40}
I20260620 19:10:02.115114 11495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:02.106032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:43665 (local address 127.0.0.1:50830)
0620 19:10:02.106365 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:02.106376 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:02.106497 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:02.106720 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:02.106722 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:02.107155 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:02.107256 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:02.107263 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:02.108010 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:02.108013 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:02.108629 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:02.108641 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:02.109057 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:02.109060 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:02.109132 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:02.109823 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:02.109836 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:02.111669 (+ 1833us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:02.113517 (+ 1848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:02.113531 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:02.113538 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:02.113813 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:02.114063 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:02.114066 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:02.114068 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:02.114140 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:02.114512 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:02.114519 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:02.114628 (+ 109us) client_negotiation.cc:769] Sending connection context
0620 19:10:02.114736 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 19:10:02.114914 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":155,"threads_started":1}
I20260620 19:10:02.120779 10952 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.9.155.126@KRBTEST.COM'} at 127.0.0.1:50830:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 19:10:02.121551 10952 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:10:02.126696 11089 tablet_service.cc:1467] Processing CreateTablet for tablet b8d158c595134ab7aea2d44a94035198 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=505f88da55b94c898331fa198dc0d9d8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 19:10:02.126696 11229 tablet_service.cc:1467] Processing CreateTablet for tablet b8d158c595134ab7aea2d44a94035198 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=505f88da55b94c898331fa198dc0d9d8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 19:10:02.126745 11355 tablet_service.cc:1467] Processing CreateTablet for tablet b8d158c595134ab7aea2d44a94035198 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=505f88da55b94c898331fa198dc0d9d8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 19:10:02.127118 11229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8d158c595134ab7aea2d44a94035198. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:02.127118 11089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8d158c595134ab7aea2d44a94035198. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:02.127117 11355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8d158c595134ab7aea2d44a94035198. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:02.130049 11458 tablet_bootstrap.cc:492] T b8d158c595134ab7aea2d44a94035198 P 3964b77bd96f435cab098d389d3b20f1: Bootstrap starting.
I20260620 19:10:02.130152 11460 tablet_bootstrap.cc:492] T b8d158c595134ab7aea2d44a94035198 P 0a8b3c70c2ec47589f5869aabbc0ce15: Bootstrap starting.
I20260620 19:10:02.130344 11459 tablet_bootstrap.cc:492] T b8d158c595134ab7aea2d44a94035198 P 72317321300c441f8fb701add3d21f06: Bootstrap starting.
I20260620 19:10:02.130970 11458 tablet_bootstrap.cc:654] T b8d158c595134ab7aea2d44a94035198 P 3964b77bd96f435cab098d389d3b20f1: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:02.131129 11460 tablet_bootstrap.cc:654] T b8d158c595134ab7aea2d44a94035198 P 0a8b3c70c2ec47589f5869aabbc0ce15: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:02.131260 11459 tablet_bootstrap.cc:654] T b8d158c595134ab7aea2d44a94035198 P 72317321300c441f8fb701add3d21f06: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:02.132140 11458 tablet_bootstrap.cc:492] T b8d158c595134ab7aea2d44a94035198 P 3964b77bd96f435cab098d389d3b20f1: No bootstrap required, opened a new log
I20260620 19:10:02.132259 11458 ts_tablet_manager.cc:1397] T b8d158c595134ab7aea2d44a94035198 P 3964b77bd96f435cab098d389d3b20f1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:10:02.132434 11459 tablet_bootstrap.cc:492] T b8d158c595134ab7aea2d44a94035198 P 72317321300c441f8fb701add3d21f06: No bootstrap required, opened a new log
I20260620 19:10:02.132534 11459 ts_tablet_manager.cc:1397] T b8d158c595134ab7aea2d44a94035198 P 72317321300c441f8fb701add3d21f06: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:10:02.132596 11460 tablet_bootstrap.cc:492] T b8d158c595134ab7aea2d44a94035198 P 0a8b3c70c2ec47589f5869aabbc0ce15: No bootstrap required, opened a new log
I20260620 19:10:02.132678 11460 ts_tablet_manager.cc:1397] T b8d158c595134ab7aea2d44a94035198 P 0a8b3c70c2ec47589f5869aabbc0ce15: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 19:10:02.132673 11458 raft_consensus.cc:348] T b8d158c595134ab7aea2d44a94035198 P 3964b77bd96f435cab098d389d3b20f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3964b77bd96f435cab098d389d3b20f1" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 34885 } } peers { permanent_uuid: "72317321300c441f8fb701add3d21f06" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 35937 } } peers { permanent_uuid: "0a8b3c70c2ec47589f5869aabbc0ce15" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 44381 } }
I20260620 19:10:02.132810 11458 raft_consensus.cc:374] T b8d158c595134ab7aea2d44a94035198 P 3964b77bd96f435cab098d389d3b20f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:02.132849 11458 raft_consensus.cc:729] T b8d158c595134ab7aea2d44a94035198 P 3964b77bd96f435cab098d389d3b20f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3964b77bd96f435cab098d389d3b20f1, State: Initialized, Role: FOLLOWER
I20260620 19:10:02.132977 11458 consensus_queue.cc:260] T b8d158c595134ab7aea2d44a94035198 P 3964b77bd96f435cab098d389d3b20f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3964b77bd96f435cab098d389d3b20f1" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 34885 } } peers { permanent_uuid: "72317321300c441f8fb701add3d21f06" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 35937 } } peers { permanent_uuid: "0a8b3c70c2ec47589f5869aabbc0ce15" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 44381 } }
I20260620 19:10:02.132973 11459 raft_consensus.cc:348] T b8d158c595134ab7aea2d44a94035198 P 72317321300c441f8fb701add3d21f06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3964b77bd96f435cab098d389d3b20f1" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 34885 } } peers { permanent_uuid: "72317321300c441f8fb701add3d21f06" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 35937 } } peers { permanent_uuid: "0a8b3c70c2ec47589f5869aabbc0ce15" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 44381 } }
I20260620 19:10:02.133081 11459 raft_consensus.cc:374] T b8d158c595134ab7aea2d44a94035198 P 72317321300c441f8fb701add3d21f06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:02.133121 11459 raft_consensus.cc:729] T b8d158c595134ab7aea2d44a94035198 P 72317321300c441f8fb701add3d21f06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72317321300c441f8fb701add3d21f06, State: Initialized, Role: FOLLOWER
I20260620 19:10:02.133078 11460 raft_consensus.cc:348] T b8d158c595134ab7aea2d44a94035198 P 0a8b3c70c2ec47589f5869aabbc0ce15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3964b77bd96f435cab098d389d3b20f1" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 34885 } } peers { permanent_uuid: "72317321300c441f8fb701add3d21f06" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 35937 } } peers { permanent_uuid: "0a8b3c70c2ec47589f5869aabbc0ce15" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 44381 } }
I20260620 19:10:02.133179 11460 raft_consensus.cc:374] T b8d158c595134ab7aea2d44a94035198 P 0a8b3c70c2ec47589f5869aabbc0ce15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:02.133219 11460 raft_consensus.cc:729] T b8d158c595134ab7aea2d44a94035198 P 0a8b3c70c2ec47589f5869aabbc0ce15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a8b3c70c2ec47589f5869aabbc0ce15, State: Initialized, Role: FOLLOWER
I20260620 19:10:02.133246 11458 ts_tablet_manager.cc:1428] T b8d158c595134ab7aea2d44a94035198 P 3964b77bd96f435cab098d389d3b20f1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:10:02.133247 11459 consensus_queue.cc:260] T b8d158c595134ab7aea2d44a94035198 P 72317321300c441f8fb701add3d21f06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3964b77bd96f435cab098d389d3b20f1" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 34885 } } peers { permanent_uuid: "72317321300c441f8fb701add3d21f06" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 35937 } } peers { permanent_uuid: "0a8b3c70c2ec47589f5869aabbc0ce15" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 44381 } }
I20260620 19:10:02.133328 11460 consensus_queue.cc:260] T b8d158c595134ab7aea2d44a94035198 P 0a8b3c70c2ec47589f5869aabbc0ce15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3964b77bd96f435cab098d389d3b20f1" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 34885 } } peers { permanent_uuid: "72317321300c441f8fb701add3d21f06" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 35937 } } peers { permanent_uuid: "0a8b3c70c2ec47589f5869aabbc0ce15" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 44381 } }
I20260620 19:10:02.133445 11464 tablet_replica.cc:439] T b8d158c595134ab7aea2d44a94035198 P 3964b77bd96f435cab098d389d3b20f1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3964b77bd96f435cab098d389d3b20f1" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 34885 } } peers { permanent_uuid: "72317321300c441f8fb701add3d21f06" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 35937 } } peers { permanent_uuid: "0a8b3c70c2ec47589f5869aabbc0ce15" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 44381 } } }
I20260620 19:10:02.133507 11459 ts_tablet_manager.cc:1428] T b8d158c595134ab7aea2d44a94035198 P 72317321300c441f8fb701add3d21f06: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:10:02.133575 11464 tablet_replica.cc:442] T b8d158c595134ab7aea2d44a94035198 P 3964b77bd96f435cab098d389d3b20f1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:10:02.133589 11460 ts_tablet_manager.cc:1428] T b8d158c595134ab7aea2d44a94035198 P 0a8b3c70c2ec47589f5869aabbc0ce15: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:10:02.133824 11465 tablet_replica.cc:439] T b8d158c595134ab7aea2d44a94035198 P 0a8b3c70c2ec47589f5869aabbc0ce15: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3964b77bd96f435cab098d389d3b20f1" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 34885 } } peers { permanent_uuid: "72317321300c441f8fb701add3d21f06" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 35937 } } peers { permanent_uuid: "0a8b3c70c2ec47589f5869aabbc0ce15" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 44381 } } }
I20260620 19:10:02.133948 11465 tablet_replica.cc:442] T b8d158c595134ab7aea2d44a94035198 P 0a8b3c70c2ec47589f5869aabbc0ce15: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:10:02.134568 11466 tablet_replica.cc:439] T b8d158c595134ab7aea2d44a94035198 P 72317321300c441f8fb701add3d21f06: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3964b77bd96f435cab098d389d3b20f1" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 34885 } } peers { permanent_uuid: "72317321300c441f8fb701add3d21f06" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 35937 } } peers { permanent_uuid: "0a8b3c70c2ec47589f5869aabbc0ce15" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 44381 } } }
I20260620 19:10:02.134691 11466 tablet_replica.cc:442] T b8d158c595134ab7aea2d44a94035198 P 72317321300c441f8fb701add3d21f06: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:10:02.200435 11466 raft_consensus.cc:482] T b8d158c595134ab7aea2d44a94035198 P 72317321300c441f8fb701add3d21f06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:10:02.200583 11466 raft_consensus.cc:504] T b8d158c595134ab7aea2d44a94035198 P 72317321300c441f8fb701add3d21f06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3964b77bd96f435cab098d389d3b20f1" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 34885 } } peers { permanent_uuid: "72317321300c441f8fb701add3d21f06" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 35937 } } peers { permanent_uuid: "0a8b3c70c2ec47589f5869aabbc0ce15" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 44381 } }
I20260620 19:10:02.201706 11466 leader_election.cc:288] T b8d158c595134ab7aea2d44a94035198 P 72317321300c441f8fb701add3d21f06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3964b77bd96f435cab098d389d3b20f1 (127.9.155.65:34885), 0a8b3c70c2ec47589f5869aabbc0ce15 (127.9.155.67:44381)
I20260620 19:10:02.206705 11500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:02.201979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.67:44381 (local address 127.9.155.66:57885)
0620 19:10:02.202457 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:02.202468 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:02.202547 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:02.202858 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:02.202861 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:02.202875 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:10:02.202960 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:02.202966 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:02.203753 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:02.203756 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:02.205226 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:02.205242 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:02.206178 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:02.206194 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:02.206332 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:02.206356 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 19:10:02.206451 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 19:10:02.206535 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"mutex_wait_us":42,"thread_start_us":103,"threads_started":1}
I20260620 19:10:02.206683 11455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:02.202078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.66:57885 (local address 127.9.155.67:44381)
0620 19:10:02.202203 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:02.202207 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:02.202542 (+ 335us) server_negotiation.cc:407] Connection header received
0620 19:10:02.202630 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:02.202633 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:02.202682 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:02.202759 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:10:02.203095 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:02.203632 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:02.205374 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:02.206001 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:02.206132 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:02.206208 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:02.206484 (+ 276us) server_negotiation.cc:299] Negotiation successful
0620 19:10:02.206566 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 19:10:02.207422 11389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8d158c595134ab7aea2d44a94035198" candidate_uuid: "72317321300c441f8fb701add3d21f06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a8b3c70c2ec47589f5869aabbc0ce15" is_pre_election: true
I20260620 19:10:02.207543 11499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:02.201953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.65:34885 (local address 127.9.155.66:52527)
0620 19:10:02.202331 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:02.202345 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:02.202457 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:02.202751 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:02.202754 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:02.202769 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:10:02.202898 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:02.202904 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:02.203919 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:02.203922 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:02.205226 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:02.205242 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:02.207101 (+ 1859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:02.207105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:02.207227 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:02.207249 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 19:10:02.207325 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 19:10:02.207394 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"mutex_wait_us":66,"thread_start_us":108,"threads_started":1}
I20260620 19:10:02.207571 11457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:02.202327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.66:52527 (local address 127.9.155.65:34885)
0620 19:10:02.202430 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:02.202434 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:02.202451 (+ 17us) server_negotiation.cc:407] Connection header received
0620 19:10:02.202620 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:02.202624 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:02.202667 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:02.202728 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:10:02.203010 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:02.203812 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:02.206272 (+ 2460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:02.206879 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:02.206995 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:02.207040 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:02.207368 (+ 328us) server_negotiation.cc:299] Negotiation successful
0620 19:10:02.207463 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 19:10:02.207588 11389 raft_consensus.cc:2454] T b8d158c595134ab7aea2d44a94035198 P 0a8b3c70c2ec47589f5869aabbc0ce15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72317321300c441f8fb701add3d21f06 in term 0.
I20260620 19:10:02.208003 11109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8d158c595134ab7aea2d44a94035198" candidate_uuid: "72317321300c441f8fb701add3d21f06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3964b77bd96f435cab098d389d3b20f1" is_pre_election: true
I20260620 19:10:02.208006 11182 leader_election.cc:302] T b8d158c595134ab7aea2d44a94035198 P 72317321300c441f8fb701add3d21f06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a8b3c70c2ec47589f5869aabbc0ce15, 72317321300c441f8fb701add3d21f06; no voters:
I20260620 19:10:02.208142 11109 raft_consensus.cc:2454] T b8d158c595134ab7aea2d44a94035198 P 3964b77bd96f435cab098d389d3b20f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72317321300c441f8fb701add3d21f06 in term 0.
I20260620 19:10:02.208272 11466 raft_consensus.cc:2781] T b8d158c595134ab7aea2d44a94035198 P 72317321300c441f8fb701add3d21f06 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:10:02.208328 11466 raft_consensus.cc:482] T b8d158c595134ab7aea2d44a94035198 P 72317321300c441f8fb701add3d21f06 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:10:02.208361 11466 raft_consensus.cc:3037] T b8d158c595134ab7aea2d44a94035198 P 72317321300c441f8fb701add3d21f06 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:02.209580 11466 raft_consensus.cc:504] T b8d158c595134ab7aea2d44a94035198 P 72317321300c441f8fb701add3d21f06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3964b77bd96f435cab098d389d3b20f1" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 34885 } } peers { permanent_uuid: "72317321300c441f8fb701add3d21f06" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 35937 } } peers { permanent_uuid: "0a8b3c70c2ec47589f5869aabbc0ce15" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 44381 } }
I20260620 19:10:02.209967 11466 leader_election.cc:288] T b8d158c595134ab7aea2d44a94035198 P 72317321300c441f8fb701add3d21f06 [CANDIDATE]: Term 1 election: Requested vote from peers 3964b77bd96f435cab098d389d3b20f1 (127.9.155.65:34885), 0a8b3c70c2ec47589f5869aabbc0ce15 (127.9.155.67:44381)
I20260620 19:10:02.210243 11109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8d158c595134ab7aea2d44a94035198" candidate_uuid: "72317321300c441f8fb701add3d21f06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3964b77bd96f435cab098d389d3b20f1"
I20260620 19:10:02.210381 11109 raft_consensus.cc:3037] T b8d158c595134ab7aea2d44a94035198 P 3964b77bd96f435cab098d389d3b20f1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:02.210495 11389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8d158c595134ab7aea2d44a94035198" candidate_uuid: "72317321300c441f8fb701add3d21f06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a8b3c70c2ec47589f5869aabbc0ce15"
I20260620 19:10:02.210618 11389 raft_consensus.cc:3037] T b8d158c595134ab7aea2d44a94035198 P 0a8b3c70c2ec47589f5869aabbc0ce15 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:02.211563 11109 raft_consensus.cc:2454] T b8d158c595134ab7aea2d44a94035198 P 3964b77bd96f435cab098d389d3b20f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72317321300c441f8fb701add3d21f06 in term 1.
I20260620 19:10:02.211702 11389 raft_consensus.cc:2454] T b8d158c595134ab7aea2d44a94035198 P 0a8b3c70c2ec47589f5869aabbc0ce15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72317321300c441f8fb701add3d21f06 in term 1.
I20260620 19:10:02.212033 11181 leader_election.cc:302] T b8d158c595134ab7aea2d44a94035198 P 72317321300c441f8fb701add3d21f06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3964b77bd96f435cab098d389d3b20f1, 72317321300c441f8fb701add3d21f06; no voters:
I20260620 19:10:02.212249 11466 raft_consensus.cc:2781] T b8d158c595134ab7aea2d44a94035198 P 72317321300c441f8fb701add3d21f06 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:10:02.212471 11466 raft_consensus.cc:686] T b8d158c595134ab7aea2d44a94035198 P 72317321300c441f8fb701add3d21f06 [term 1 LEADER]: Becoming Leader. State: Replica: 72317321300c441f8fb701add3d21f06, State: Running, Role: LEADER
I20260620 19:10:02.212706 11466 consensus_queue.cc:237] T b8d158c595134ab7aea2d44a94035198 P 72317321300c441f8fb701add3d21f06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3964b77bd96f435cab098d389d3b20f1" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 34885 } } peers { permanent_uuid: "72317321300c441f8fb701add3d21f06" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 35937 } } peers { permanent_uuid: "0a8b3c70c2ec47589f5869aabbc0ce15" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 44381 } }
I20260620 19:10:02.213853 11501 tablet_replica.cc:439] T b8d158c595134ab7aea2d44a94035198 P 72317321300c441f8fb701add3d21f06: TxnStatusTablet state changed. Reason: New leader 72317321300c441f8fb701add3d21f06. Latest consensus state: current_term: 1 leader_uuid: "72317321300c441f8fb701add3d21f06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3964b77bd96f435cab098d389d3b20f1" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 34885 } } peers { permanent_uuid: "72317321300c441f8fb701add3d21f06" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 35937 } } peers { permanent_uuid: "0a8b3c70c2ec47589f5869aabbc0ce15" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 44381 } } }
I20260620 19:10:02.213966 11501 tablet_replica.cc:442] T b8d158c595134ab7aea2d44a94035198 P 72317321300c441f8fb701add3d21f06: This TxnStatusTablet replica's current role is: LEADER
I20260620 19:10:02.214221 11503 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 19:10:02.214382 11503 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 19:10:02.215008 10951 catalog_manager.cc:5495] T b8d158c595134ab7aea2d44a94035198 P 72317321300c441f8fb701add3d21f06 reported cstate change: term changed from 0 to 1, leader changed from <none> to 72317321300c441f8fb701add3d21f06 (127.9.155.66). New cstate: current_term: 1 leader_uuid: "72317321300c441f8fb701add3d21f06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3964b77bd96f435cab098d389d3b20f1" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 34885 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72317321300c441f8fb701add3d21f06" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 35937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a8b3c70c2ec47589f5869aabbc0ce15" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 44381 } health_report { overall_health: UNKNOWN } } }
I20260620 19:10:02.240542 11495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:02.236062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.66:35937 (local address 127.0.0.1:48174)
0620 19:10:02.236333 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:02.236346 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:02.236438 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:02.236791 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:02.236794 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:02.236802 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 19:10:02.236900 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:02.236905 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:02.237900 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:02.237903 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:02.238539 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:02.238547 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:02.239191 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:02.239195 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:02.239281 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:02.239330 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 19:10:02.240171 (+ 841us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 19:10:02.240189 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 19:10:02.240335 (+ 146us) client_negotiation.cc:240] Negotiation successful
0620 19:10:02.240421 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168}
I20260620 19:10:02.240582 11456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:02.236175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48174 (local address 127.9.155.66:35937)
0620 19:10:02.236309 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:02.236313 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:02.236400 (+ 87us) server_negotiation.cc:407] Connection header received
0620 19:10:02.236565 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:02.236569 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:02.236637 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:02.236742 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:10:02.237007 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:02.237786 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:02.238646 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:02.239057 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:02.239164 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:02.239485 (+ 321us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:10:02.240064 (+ 579us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:10:02.240176 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:02.240360 (+ 184us) server_negotiation.cc:299] Negotiation successful
0620 19:10:02.240425 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 19:10:02.243525 11389 raft_consensus.cc:1261] T b8d158c595134ab7aea2d44a94035198 P 0a8b3c70c2ec47589f5869aabbc0ce15 [term 1 FOLLOWER]: Refusing update from remote peer 72317321300c441f8fb701add3d21f06: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:10:02.243695 11109 raft_consensus.cc:1261] T b8d158c595134ab7aea2d44a94035198 P 3964b77bd96f435cab098d389d3b20f1 [term 1 FOLLOWER]: Refusing update from remote peer 72317321300c441f8fb701add3d21f06: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:10:02.244130 11501 consensus_queue.cc:1035] T b8d158c595134ab7aea2d44a94035198 P 72317321300c441f8fb701add3d21f06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a8b3c70c2ec47589f5869aabbc0ce15" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 44381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:10:02.244335 11466 consensus_queue.cc:1035] T b8d158c595134ab7aea2d44a94035198 P 72317321300c441f8fb701add3d21f06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3964b77bd96f435cab098d389d3b20f1" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 34885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:10:02.245888 11465 tablet_replica.cc:439] T b8d158c595134ab7aea2d44a94035198 P 0a8b3c70c2ec47589f5869aabbc0ce15: TxnStatusTablet state changed. Reason: New leader 72317321300c441f8fb701add3d21f06. Latest consensus state: current_term: 1 leader_uuid: "72317321300c441f8fb701add3d21f06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3964b77bd96f435cab098d389d3b20f1" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 34885 } } peers { permanent_uuid: "72317321300c441f8fb701add3d21f06" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 35937 } } peers { permanent_uuid: "0a8b3c70c2ec47589f5869aabbc0ce15" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 44381 } } }
I20260620 19:10:02.245898 11464 tablet_replica.cc:439] T b8d158c595134ab7aea2d44a94035198 P 3964b77bd96f435cab098d389d3b20f1: TxnStatusTablet state changed. Reason: New leader 72317321300c441f8fb701add3d21f06. Latest consensus state: current_term: 1 leader_uuid: "72317321300c441f8fb701add3d21f06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3964b77bd96f435cab098d389d3b20f1" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 34885 } } peers { permanent_uuid: "72317321300c441f8fb701add3d21f06" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 35937 } } peers { permanent_uuid: "0a8b3c70c2ec47589f5869aabbc0ce15" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 44381 } } }
I20260620 19:10:02.246011 11465 tablet_replica.cc:442] T b8d158c595134ab7aea2d44a94035198 P 0a8b3c70c2ec47589f5869aabbc0ce15: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:10:02.246014 11464 tablet_replica.cc:442] T b8d158c595134ab7aea2d44a94035198 P 3964b77bd96f435cab098d389d3b20f1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:10:02.248983 11472 mvcc.cc:205] Tried to move back new op lower bound from 7299000738784219136 to 7299000738665553920. Current Snapshot: MvccSnapshot[applied={T|T < 7299000738784219136}]
I20260620 19:10:02.249331 11464 tablet_replica.cc:439] T b8d158c595134ab7aea2d44a94035198 P 3964b77bd96f435cab098d389d3b20f1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "72317321300c441f8fb701add3d21f06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3964b77bd96f435cab098d389d3b20f1" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 34885 } } peers { permanent_uuid: "72317321300c441f8fb701add3d21f06" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 35937 } } peers { permanent_uuid: "0a8b3c70c2ec47589f5869aabbc0ce15" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 44381 } } }
I20260620 19:10:02.249426 11464 tablet_replica.cc:442] T b8d158c595134ab7aea2d44a94035198 P 3964b77bd96f435cab098d389d3b20f1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:10:02.250247 11466 tablet_replica.cc:439] T b8d158c595134ab7aea2d44a94035198 P 72317321300c441f8fb701add3d21f06: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "72317321300c441f8fb701add3d21f06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3964b77bd96f435cab098d389d3b20f1" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 34885 } } peers { permanent_uuid: "72317321300c441f8fb701add3d21f06" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 35937 } } peers { permanent_uuid: "0a8b3c70c2ec47589f5869aabbc0ce15" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 44381 } } }
I20260620 19:10:02.250288 11465 tablet_replica.cc:439] T b8d158c595134ab7aea2d44a94035198 P 0a8b3c70c2ec47589f5869aabbc0ce15: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "72317321300c441f8fb701add3d21f06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3964b77bd96f435cab098d389d3b20f1" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 34885 } } peers { permanent_uuid: "72317321300c441f8fb701add3d21f06" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 35937 } } peers { permanent_uuid: "0a8b3c70c2ec47589f5869aabbc0ce15" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 44381 } } }
I20260620 19:10:02.250357 11466 tablet_replica.cc:442] T b8d158c595134ab7aea2d44a94035198 P 72317321300c441f8fb701add3d21f06: This TxnStatusTablet replica's current role is: LEADER
I20260620 19:10:02.250372 11465 tablet_replica.cc:442] T b8d158c595134ab7aea2d44a94035198 P 0a8b3c70c2ec47589f5869aabbc0ce15: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:10:02.250461 11466 tablet_replica.cc:439] T b8d158c595134ab7aea2d44a94035198 P 72317321300c441f8fb701add3d21f06: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "72317321300c441f8fb701add3d21f06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3964b77bd96f435cab098d389d3b20f1" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 34885 } } peers { permanent_uuid: "72317321300c441f8fb701add3d21f06" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 35937 } } peers { permanent_uuid: "0a8b3c70c2ec47589f5869aabbc0ce15" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 44381 } } }
I20260620 19:10:02.250524 11466 tablet_replica.cc:442] T b8d158c595134ab7aea2d44a94035198 P 72317321300c441f8fb701add3d21f06: This TxnStatusTablet replica's current role is: LEADER
I20260620 19:10:02.258049 11455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:02.254363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58568 (local address 127.9.155.67:44381)
0620 19:10:02.254464 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:02.254467 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:02.254479 (+ 12us) server_negotiation.cc:407] Connection header received
0620 19:10:02.254543 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:02.254545 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:02.254594 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:02.254675 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:10:02.254907 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:02.255426 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:02.256575 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:02.256821 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:02.256906 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:02.257183 (+ 277us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:10:02.257615 (+ 432us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:10:02.257797 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:02.257857 (+ 60us) server_negotiation.cc:299] Negotiation successful
0620 19:10:02.257908 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 19:10:02.268774 11456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:02.265089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.67:59083 (local address 127.9.155.66:35937)
0620 19:10:02.265214 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:02.265218 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:02.265363 (+ 145us) server_negotiation.cc:407] Connection header received
0620 19:10:02.265488 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:02.265490 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:02.265535 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:02.265596 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:10:02.265838 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:02.266360 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:02.267604 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:02.267873 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:02.267961 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:02.268341 (+ 380us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:10:02.268430 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:10:02.268498 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:02.268593 (+ 95us) server_negotiation.cc:299] Negotiation successful
0620 19:10:02.268642 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 19:10:02.268842 11511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:02.264971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.66:35937 (local address 127.9.155.67:59083)
0620 19:10:02.265281 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:02.265296 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:02.265402 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:02.265615 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:02.265618 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:02.265625 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 19:10:02.265754 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:02.265760 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:02.266457 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:02.266461 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:02.267455 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:02.267465 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:02.267981 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:02.267985 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:02.268086 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:02.268144 (+ 58us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 19:10:02.268512 (+ 368us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 19:10:02.268518 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 19:10:02.268580 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 19:10:02.268666 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":114,"threads_started":1}
I20260620 19:10:02.276616 11249 raft_consensus.cc:1261] T fb9d48d591fb45308d690147e79944e7 P 72317321300c441f8fb701add3d21f06 [term 1 FOLLOWER]: Refusing update from remote peer 0a8b3c70c2ec47589f5869aabbc0ce15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:10:02.276616 11109 raft_consensus.cc:1261] T fb9d48d591fb45308d690147e79944e7 P 3964b77bd96f435cab098d389d3b20f1 [term 1 FOLLOWER]: Refusing update from remote peer 0a8b3c70c2ec47589f5869aabbc0ce15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:10:02.277108 11488 consensus_queue.cc:1035] T fb9d48d591fb45308d690147e79944e7 P 0a8b3c70c2ec47589f5869aabbc0ce15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72317321300c441f8fb701add3d21f06" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 35937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:10:02.277352 11465 consensus_queue.cc:1035] T fb9d48d591fb45308d690147e79944e7 P 0a8b3c70c2ec47589f5869aabbc0ce15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3964b77bd96f435cab098d389d3b20f1" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 34885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:10:02.300506 11517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:02.296297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.67:44381 (local address 127.9.155.66:60637)
0620 19:10:02.296701 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:02.296714 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:02.296871 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:02.297086 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:02.297089 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:02.297096 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 19:10:02.297214 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:02.297220 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:02.298230 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:02.298233 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:02.299208 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:02.299218 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:02.299790 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:02.299792 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:02.299868 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:02.299912 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 19:10:02.300179 (+ 267us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 19:10:02.300182 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:10:02.300250 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 19:10:02.300375 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":112,"threads_started":1}
I20260620 19:10:02.300472 11455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:02.296344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.66:60637 (local address 127.9.155.67:44381)
0620 19:10:02.296452 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:02.296457 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:02.296801 (+ 344us) server_negotiation.cc:407] Connection header received
0620 19:10:02.296966 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:02.296969 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:02.297015 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:02.297071 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:10:02.297325 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:02.298133 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:02.299335 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:02.299681 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:02.299791 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:02.300028 (+ 237us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:10:02.300108 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:10:02.300179 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:02.300259 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 19:10:02.300314 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 19:10:02.327168 10952 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50826:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:10:02.327332 10952 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50826:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:10:02.329571 10952 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 05b0609251bd43e5aeb2a0b1a38f4f49: Sending DeleteTablet for 3 replicas of tablet fb9d48d591fb45308d690147e79944e7
I20260620 19:10:02.330176 11089 tablet_service.cc:1514] Processing DeleteTablet for tablet fb9d48d591fb45308d690147e79944e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:10:02 UTC) from {username='kudu', principal='kudu/127.9.155.126@KRBTEST.COM'} at 127.0.0.1:37370
I20260620 19:10:02.330391 11483 tablet_replica.cc:330] T fb9d48d591fb45308d690147e79944e7 P 3964b77bd96f435cab098d389d3b20f1: stopping tablet replica
I20260620 19:10:02.330379 11229 tablet_service.cc:1514] Processing DeleteTablet for tablet fb9d48d591fb45308d690147e79944e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:10:02 UTC) from {username='kudu', principal='kudu/127.9.155.126@KRBTEST.COM'} at 127.0.0.1:48172
I20260620 19:10:02.330510 11483 raft_consensus.cc:2229] T fb9d48d591fb45308d690147e79944e7 P 3964b77bd96f435cab098d389d3b20f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:10:02.330636 11483 raft_consensus.cc:2258] T fb9d48d591fb45308d690147e79944e7 P 3964b77bd96f435cab098d389d3b20f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:10:02.330802 11482 tablet_replica.cc:330] T fb9d48d591fb45308d690147e79944e7 P 72317321300c441f8fb701add3d21f06: stopping tablet replica
I20260620 19:10:02.331027 11355 tablet_service.cc:1514] Processing DeleteTablet for tablet fb9d48d591fb45308d690147e79944e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:10:02 UTC) from {username='kudu', principal='kudu/127.9.155.126@KRBTEST.COM'} at 127.0.0.1:58558
I20260620 19:10:02.331188 11481 tablet_replica.cc:330] T fb9d48d591fb45308d690147e79944e7 P 0a8b3c70c2ec47589f5869aabbc0ce15: stopping tablet replica
I20260620 19:10:02.331285 11481 raft_consensus.cc:2229] T fb9d48d591fb45308d690147e79944e7 P 0a8b3c70c2ec47589f5869aabbc0ce15 [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:10:02.331343 11482 raft_consensus.cc:2229] T fb9d48d591fb45308d690147e79944e7 P 72317321300c441f8fb701add3d21f06 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:10:02.331465 11482 raft_consensus.cc:2258] T fb9d48d591fb45308d690147e79944e7 P 72317321300c441f8fb701add3d21f06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:10:02.331585 11481 raft_consensus.cc:2258] T fb9d48d591fb45308d690147e79944e7 P 0a8b3c70c2ec47589f5869aabbc0ce15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:10:02.332372 11481 ts_tablet_manager.cc:1905] T fb9d48d591fb45308d690147e79944e7 P 0a8b3c70c2ec47589f5869aabbc0ce15: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:10:02.332373 11483 ts_tablet_manager.cc:1905] T fb9d48d591fb45308d690147e79944e7 P 3964b77bd96f435cab098d389d3b20f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:10:02.335191 11483 ts_tablet_manager.cc:1918] T fb9d48d591fb45308d690147e79944e7 P 3964b77bd96f435cab098d389d3b20f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 19:10:02.335191 11481 ts_tablet_manager.cc:1918] T fb9d48d591fb45308d690147e79944e7 P 0a8b3c70c2ec47589f5869aabbc0ce15: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 19:10:02.335261 11481 log.cc:1196] T fb9d48d591fb45308d690147e79944e7 P 0a8b3c70c2ec47589f5869aabbc0ce15: Deleting WAL directory at /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/minicluster-data/ts-2/wal/wals/fb9d48d591fb45308d690147e79944e7
I20260620 19:10:02.335261 11483 log.cc:1196] T fb9d48d591fb45308d690147e79944e7 P 3964b77bd96f435cab098d389d3b20f1: Deleting WAL directory at /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/minicluster-data/ts-0/wal/wals/fb9d48d591fb45308d690147e79944e7
I20260620 19:10:02.335490 11483 ts_tablet_manager.cc:1939] T fb9d48d591fb45308d690147e79944e7 P 3964b77bd96f435cab098d389d3b20f1: Deleting consensus metadata
I20260620 19:10:02.335490 11481 ts_tablet_manager.cc:1939] T fb9d48d591fb45308d690147e79944e7 P 0a8b3c70c2ec47589f5869aabbc0ce15: Deleting consensus metadata
I20260620 19:10:02.336442 10938 catalog_manager.cc:4841] TS 0a8b3c70c2ec47589f5869aabbc0ce15 (127.9.155.67:44381): tablet fb9d48d591fb45308d690147e79944e7 (table test-table [id=ca9eed32276e4845b7446e76421cce4a]) successfully deleted
I20260620 19:10:02.336517 11482 ts_tablet_manager.cc:1905] T fb9d48d591fb45308d690147e79944e7 P 72317321300c441f8fb701add3d21f06: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:10:02.338639 11482 ts_tablet_manager.cc:1918] T fb9d48d591fb45308d690147e79944e7 P 72317321300c441f8fb701add3d21f06: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 19:10:02.338692 11482 log.cc:1196] T fb9d48d591fb45308d690147e79944e7 P 72317321300c441f8fb701add3d21f06: Deleting WAL directory at /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982595231230-9837-0/minicluster-data/ts-1/wal/wals/fb9d48d591fb45308d690147e79944e7
I20260620 19:10:02.338856 11482 ts_tablet_manager.cc:1939] T fb9d48d591fb45308d690147e79944e7 P 72317321300c441f8fb701add3d21f06: Deleting consensus metadata
I20260620 19:10:02.339524 10939 catalog_manager.cc:4841] TS 72317321300c441f8fb701add3d21f06 (127.9.155.66:35937): tablet fb9d48d591fb45308d690147e79944e7 (table test-table [id=ca9eed32276e4845b7446e76421cce4a]) successfully deleted
I20260620 19:10:02.341012 10937 catalog_manager.cc:4841] TS 3964b77bd96f435cab098d389d3b20f1 (127.9.155.65:34885): tablet fb9d48d591fb45308d690147e79944e7 (table test-table [id=ca9eed32276e4845b7446e76421cce4a]) successfully deleted
Jun 20 19:10:02 dist-test-slave-n3fx krb5kdc[10907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982601, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.155.65@KRBTEST.COM
I20260620 19:10:02.350734 11457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:02.340805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37376 (local address 127.9.155.65:34885)
0620 19:10:02.340921 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:02.340925 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:02.341428 (+ 503us) server_negotiation.cc:407] Connection header received
0620 19:10:02.341536 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:02.341539 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:02.341584 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:02.341649 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:02.342330 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:02.342830 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:02.343623 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:02.343851 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:02.343939 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:02.347083 (+ 3144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:02.347097 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:02.347114 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:02.347160 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:02.348792 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:02.349208 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:02.349214 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:02.349217 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:02.349297 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:02.349545 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:02.349549 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:02.349551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:02.349819 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:02.350012 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:02.350238 (+ 226us) server_negotiation.cc:299] Negotiation successful
0620 19:10:02.350555 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260620 19:10:02.352010 11149 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37376
I20260620 19:10:02.363387 11001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:02.354063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50844 (local address 127.9.155.126:43665)
0620 19:10:02.354186 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:02.354191 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:02.354627 (+ 436us) server_negotiation.cc:407] Connection header received
0620 19:10:02.354732 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:02.354735 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:02.354790 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:02.354868 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:02.355624 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:02.356460 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:02.357585 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:02.357967 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:02.358083 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:02.359749 (+ 1666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:02.359753 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:02.359757 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:02.359787 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:02.361477 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:02.361961 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:02.361965 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:02.361967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:02.362067 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:02.362340 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:02.362344 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:02.362346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:02.362598 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:02.362756 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:02.363042 (+ 286us) server_negotiation.cc:299] Negotiation successful
0620 19:10:02.363214 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260620 19:10:02.363998 10952 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50844
I20260620 19:10:02.365424 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 11024
I20260620 19:10:02.374694 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 11164
I20260620 19:10:02.381654 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 11304
I20260620 19:10:02.387022 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 10920
2026-06-20T19:10:02Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2755 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:10:02 dist-test-slave-n3fx krb5kdc[11526](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:10:02 dist-test-slave-n3fx krb5kdc[11526](info): set up 2 sockets
Jun 20 19:10:02 dist-test-slave-n3fx krb5kdc[11526](info): commencing operation
krb5kdc: starting...
W20260620 19:10:03.351855 9837 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/krb5kdc/test-user.keytab.
Jun 20 19:10:03 dist-test-slave-n3fx krb5kdc[11526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982603, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:10:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:10:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.155.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.155.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.155.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab.
Entry for principal kudu/127.9.155.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab.
I20260620 19:10:03.471556 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:37761
--webserver_interface=127.9.155.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab
--principal=kudu/127.9.155.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:34135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.155.126:37761
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:10:03.581434 11539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:03.581732 11539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:03.581790 11539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:03.585924 11539 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:10:03.585996 11539 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 19:10:03.586018 11539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:03.586042 11539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:03.586064 11539 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:10:03.586092 11539 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:10:03.591586 11539 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:34135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.155.126:37761
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab
--principal=kudu/127.9.155.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:37761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.155.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11539
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:10:03.592671 11539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:03.593677 11539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:10:03.601874 11547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:03.601899 11544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:03.601876 11545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:03.602113 11539 server_base.cc:1034] running on GCE node
I20260620 19:10:03.602735 11539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:03.603785 11539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:03.604957 11539 hybrid_clock.cc:648] HybridClock initialized: now 1781982603604943 us; error 28 us; skew 500 ppm
Jun 20 19:10:03 dist-test-slave-n3fx krb5kdc[11526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982603, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:03.610482 11539 init.cc:377] Logged in from keytab as kudu/127.9.155.126@KRBTEST.COM (short username kudu)
I20260620 19:10:03.611450 11539 webserver.cc:460] Webserver started at http://127.9.155.126:41025/ using document root <none> and password file <none>
I20260620 19:10:03.612035 11539 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:03.612105 11539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:03.612334 11539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:10:03.614411 11539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/minicluster-data/master-0/data/instance:
uuid: "4d94d0235626484baae6753c95f8e3cf"
format_stamp: "Formatted at 2026-06-20 19:10:03 on dist-test-slave-n3fx"
server_key: "41c36d90616929dfc72da8d3132a09a6"
server_key_iv: "878633691dc09f18f6aeab5b29561e3b"
server_key_version: "encryptionkey@0"
I20260620 19:10:03.614965 11539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/minicluster-data/master-0/wal/instance:
uuid: "4d94d0235626484baae6753c95f8e3cf"
format_stamp: "Formatted at 2026-06-20 19:10:03 on dist-test-slave-n3fx"
server_key: "41c36d90616929dfc72da8d3132a09a6"
server_key_iv: "878633691dc09f18f6aeab5b29561e3b"
server_key_version: "encryptionkey@0"
I20260620 19:10:03.618758 11539 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260620 19:10:03.621248 11554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:03.622577 11539 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:10:03.622695 11539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/minicluster-data/master-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/minicluster-data/master-0/wal
uuid: "4d94d0235626484baae6753c95f8e3cf"
format_stamp: "Formatted at 2026-06-20 19:10:03 on dist-test-slave-n3fx"
server_key: "41c36d90616929dfc72da8d3132a09a6"
server_key_iv: "878633691dc09f18f6aeab5b29561e3b"
server_key_version: "encryptionkey@0"
I20260620 19:10:03.622805 11539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:03.639024 11539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:03.642114 11539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:03.642305 11539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:03.649809 11539 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.126:37761
I20260620 19:10:03.649822 11616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.126:37761 every 8 connection(s)
I20260620 19:10:03.650851 11539 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
I20260620 19:10:03.653683 11617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:03.657980 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 11539
I20260620 19:10:03.658082 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/minicluster-data/master-0/wal/instance
I20260620 19:10:03.658324 9837 external_mini_cluster.cc:1428] Setting key 6be947ba4b4303f5ed0782f93900238c
I20260620 19:10:03.659377 11617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d94d0235626484baae6753c95f8e3cf: Bootstrap starting.
I20260620 19:10:03.661821 11617 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d94d0235626484baae6753c95f8e3cf: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:03.662577 11617 log.cc:824] T 00000000000000000000000000000000 P 4d94d0235626484baae6753c95f8e3cf: Log is configured to *not* fsync() on all Append() calls
I20260620 19:10:03.664558 11617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d94d0235626484baae6753c95f8e3cf: No bootstrap required, opened a new log
Jun 20 19:10:03 dist-test-slave-n3fx krb5kdc[11526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982603, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:10:03.668767 11617 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d94d0235626484baae6753c95f8e3cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d94d0235626484baae6753c95f8e3cf" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 37761 } }
I20260620 19:10:03.669081 11617 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d94d0235626484baae6753c95f8e3cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:03.669168 11617 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d94d0235626484baae6753c95f8e3cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d94d0235626484baae6753c95f8e3cf, State: Initialized, Role: FOLLOWER
I20260620 19:10:03.669746 11617 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d94d0235626484baae6753c95f8e3cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d94d0235626484baae6753c95f8e3cf" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 37761 } }
I20260620 19:10:03.669914 11617 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d94d0235626484baae6753c95f8e3cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:10:03.669981 11617 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d94d0235626484baae6753c95f8e3cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:10:03.670076 11617 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d94d0235626484baae6753c95f8e3cf [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:03.671176 11617 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d94d0235626484baae6753c95f8e3cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d94d0235626484baae6753c95f8e3cf" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 37761 } }
I20260620 19:10:03.671571 11617 leader_election.cc:302] T 00000000000000000000000000000000 P 4d94d0235626484baae6753c95f8e3cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d94d0235626484baae6753c95f8e3cf; no voters:
I20260620 19:10:03.671916 11617 leader_election.cc:288] T 00000000000000000000000000000000 P 4d94d0235626484baae6753c95f8e3cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:10:03.672060 11622 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d94d0235626484baae6753c95f8e3cf [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:10:03.672302 11622 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d94d0235626484baae6753c95f8e3cf [term 1 LEADER]: Becoming Leader. State: Replica: 4d94d0235626484baae6753c95f8e3cf, State: Running, Role: LEADER
I20260620 19:10:03.672760 11622 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d94d0235626484baae6753c95f8e3cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d94d0235626484baae6753c95f8e3cf" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 37761 } }
I20260620 19:10:03.673285 11617 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d94d0235626484baae6753c95f8e3cf [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:10:03.673761 11620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:03.659752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53076 (local address 127.9.155.126:37761)
0620 19:10:03.660338 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:03.660356 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:03.660396 (+ 40us) server_negotiation.cc:407] Connection header received
0620 19:10:03.661070 (+ 674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:03.661089 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:03.661478 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:03.661901 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:03.662767 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:03.663800 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:03.664719 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:03.665402 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:03.665580 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:03.668240 (+ 2660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:03.668247 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:03.668267 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:03.668300 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:03.670353 (+ 2053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:03.670778 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:03.670784 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:03.670790 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:03.670952 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:03.671209 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:03.671213 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:03.671214 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:03.671749 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:03.671988 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:03.672480 (+ 492us) server_negotiation.cc:299] Negotiation successful
0620 19:10:03.672778 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":133,"threads_started":1}
I20260620 19:10:03.673866 11622 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d94d0235626484baae6753c95f8e3cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d94d0235626484baae6753c95f8e3cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d94d0235626484baae6753c95f8e3cf" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 37761 } } }
I20260620 19:10:03.673977 11622 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d94d0235626484baae6753c95f8e3cf [sys.catalog]: This master's current role is: LEADER
I20260620 19:10:03.674178 11622 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d94d0235626484baae6753c95f8e3cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d94d0235626484baae6753c95f8e3cf. Latest consensus state: current_term: 1 leader_uuid: "4d94d0235626484baae6753c95f8e3cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d94d0235626484baae6753c95f8e3cf" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 37761 } } }
I20260620 19:10:03.674257 11622 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d94d0235626484baae6753c95f8e3cf [sys.catalog]: This master's current role is: LEADER
I20260620 19:10:03.674569 11628 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:10:03.677927 11628 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:10:03.684473 11628 catalog_manager.cc:1347] Generated new cluster ID: a54e35edd8054a3392439c7f64940942
I20260620 19:10:03.684569 11628 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:10:03.692509 11628 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:10:03.693542 11628 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:10:03.703764 11628 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d94d0235626484baae6753c95f8e3cf: Generated new TSK 0
I20260620 19:10:03.704699 11628 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.155.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.155.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.155.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.65.keytab.
Entry for principal kudu/127.9.155.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.65.keytab.
I20260620 19:10:03.735716 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.65:0
--local_ip_for_outbound_sockets=127.9.155.65
--webserver_interface=127.9.155.65
--webserver_port=0
--tserver_master_addrs=127.9.155.126:37761
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.65.keytab
--principal=kudu/127.9.155.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:34135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:10:03.865835 11643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:03.866106 11643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:03.866159 11643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:03.870354 11643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:03.870446 11643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:03.870496 11643 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 19:10:03.870509 11643 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 19:10:03.870549 11643 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.65
I20260620 19:10:03.876278 11643 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:34135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.65.keytab
--principal=kudu/127.9.155.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.155.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.155.126:37761
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.65
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:10:03.877393 11643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:03.878407 11643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:10:03.888032 11648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:03.888072 11643 server_base.cc:1034] running on GCE node
W20260620 19:10:03.888046 11649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:03.888334 11651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:03.888732 11643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:03.889385 11643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:03.890544 11643 hybrid_clock.cc:648] HybridClock initialized: now 1781982603890523 us; error 36 us; skew 500 ppm
Jun 20 19:10:03 dist-test-slave-n3fx krb5kdc[11526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982603, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:03.893254 11643 init.cc:377] Logged in from keytab as kudu/127.9.155.65@KRBTEST.COM (short username kudu)
I20260620 19:10:03.894358 11643 webserver.cc:460] Webserver started at http://127.9.155.65:37107/ using document root <none> and password file <none>
I20260620 19:10:03.894905 11643 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:03.894973 11643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:03.895159 11643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:10:03.897245 11643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/minicluster-data/ts-0/data/instance:
uuid: "be1d9954fe514ae18445ac0250dae2d8"
format_stamp: "Formatted at 2026-06-20 19:10:03 on dist-test-slave-n3fx"
server_key: "66e8fc9b36f4a7bb260b8e900e77a8a8"
server_key_iv: "70ca5825ffefce9cbc9936b20a4e4dc8"
server_key_version: "encryptionkey@0"
I20260620 19:10:03.897794 11643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/minicluster-data/ts-0/wal/instance:
uuid: "be1d9954fe514ae18445ac0250dae2d8"
format_stamp: "Formatted at 2026-06-20 19:10:03 on dist-test-slave-n3fx"
server_key: "66e8fc9b36f4a7bb260b8e900e77a8a8"
server_key_iv: "70ca5825ffefce9cbc9936b20a4e4dc8"
server_key_version: "encryptionkey@0"
I20260620 19:10:03.901466 11643 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 19:10:03.903946 11658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:03.905251 11643 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 19:10:03.905397 11643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/minicluster-data/ts-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/minicluster-data/ts-0/wal
uuid: "be1d9954fe514ae18445ac0250dae2d8"
format_stamp: "Formatted at 2026-06-20 19:10:03 on dist-test-slave-n3fx"
server_key: "66e8fc9b36f4a7bb260b8e900e77a8a8"
server_key_iv: "70ca5825ffefce9cbc9936b20a4e4dc8"
server_key_version: "encryptionkey@0"
I20260620 19:10:03.905499 11643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:03.919603 11643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:03.923017 11643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:03.923204 11643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:03.924930 11643 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:10:03.924988 11643 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:03.925042 11643 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:10:03.925071 11643 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:03.935343 11643 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.65:43623
Jun 20 19:10:03 dist-test-slave-n3fx krb5kdc[11526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982603, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.65@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:10:03.935408 11773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.65:43623 every 8 connection(s)
I20260620 19:10:03.936286 11643 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
I20260620 19:10:03.939659 11620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:03.926151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.65:37999 (local address 127.9.155.126:37761)
0620 19:10:03.926348 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:03.926352 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:03.927289 (+ 937us) server_negotiation.cc:407] Connection header received
0620 19:10:03.928482 (+ 1193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:03.928486 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:03.928544 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:03.928621 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:03.930174 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:03.930747 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:03.932432 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:03.932736 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:03.932828 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:03.936072 (+ 3244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:03.936076 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:03.936079 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:03.936111 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:03.937693 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:03.938230 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:03.938233 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:03.938234 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:03.938313 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:03.938641 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:03.938643 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:03.938645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:03.938824 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:03.938894 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:03.939420 (+ 526us) server_negotiation.cc:299] Negotiation successful
0620 19:10:03.939533 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260620 19:10:03.940541 11667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:03.926638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:37761 (local address 127.9.155.65:37999)
0620 19:10:03.927137 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:03.927179 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:03.928245 (+ 1066us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:03.928835 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:03.928844 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:03.929305 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:03.930018 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:03.930039 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:03.930872 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:03.930876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:03.932303 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:03.932316 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:03.932913 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:03.932917 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:03.933118 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:03.933833 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:03.933858 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:03.935884 (+ 2026us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:03.937810 (+ 1926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:03.937816 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:03.937831 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:03.938145 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:03.938432 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:03.938434 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:03.938436 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:03.938573 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:03.938899 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:03.938904 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:03.939190 (+ 286us) client_negotiation.cc:769] Sending connection context
0620 19:10:03.939370 (+ 180us) client_negotiation.cc:240] Negotiation successful
0620 19:10:03.939646 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":132,"threads_started":1}
I20260620 19:10:03.941840 11774 heartbeater.cc:347] Connected to a master server at 127.9.155.126:37761
I20260620 19:10:03.941983 11774 heartbeater.cc:464] Registering TS with master...
I20260620 19:10:03.942471 11774 heartbeater.cc:511] Master 127.9.155.126:37761 requested a full tablet report, sending...
I20260620 19:10:03.942976 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 11643
I20260620 19:10:03.943095 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/minicluster-data/ts-0/wal/instance
I20260620 19:10:03.943389 9837 external_mini_cluster.cc:1428] Setting key 4cc2d6b11cde8d910c21a4ba245d8282
I20260620 19:10:03.944180 11571 ts_manager.cc:194] Registered new tserver with Master: be1d9954fe514ae18445ac0250dae2d8 (127.9.155.65:43623)
I20260620 19:10:03.945385 11571 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.155.65@KRBTEST.COM'} at 127.9.155.65:37999
WARNING: no policy specified for kudu/127.9.155.66@KRBTEST.COM; defaulting to no policy
I20260620 19:10:03.955250 11780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:03.944057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:37761 (local address 127.9.155.65:36581)
0620 19:10:03.946054 (+ 1997us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:03.946071 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:03.946179 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:03.948393 (+ 2214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:03.948396 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:03.948616 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:03.948745 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:03.948753 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:03.949579 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:03.949582 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:03.950470 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:03.950482 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:03.950944 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:03.950948 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:03.951044 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:03.951573 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:03.951589 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:03.952015 (+ 426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:03.953777 (+ 1762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:03.953781 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:03.953782 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:03.954003 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:03.954271 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:03.954273 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:03.954275 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:03.954379 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:03.954794 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:03.954797 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:03.954882 (+ 85us) client_negotiation.cc:769] Sending connection context
0620 19:10:03.954964 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 19:10:03.955084 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1904,"thread_start_us":95,"threads_started":1}
I20260620 19:10:03.955265 11620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:03.944116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.65:36581 (local address 127.9.155.126:37761)
0620 19:10:03.944250 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:03.944254 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:03.946147 (+ 1893us) server_negotiation.cc:407] Connection header received
0620 19:10:03.946312 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:03.946316 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:03.946360 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:03.946400 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:03.948879 (+ 2479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:03.949444 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:03.950599 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:03.950825 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:03.950904 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:03.952260 (+ 1356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:03.952264 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:03.952267 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:03.952301 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:03.953675 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:03.954111 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:03.954114 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:03.954115 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:03.954196 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:03.954484 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:03.954487 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:03.954488 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:03.954700 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:03.954797 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:03.955035 (+ 238us) server_negotiation.cc:299] Negotiation successful
0620 19:10:03.955139 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.155.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.155.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.66.keytab.
Entry for principal kudu/127.9.155.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.66.keytab.
I20260620 19:10:03.971208 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.66:0
--local_ip_for_outbound_sockets=127.9.155.66
--webserver_interface=127.9.155.66
--webserver_port=0
--tserver_master_addrs=127.9.155.126:37761
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.66.keytab
--principal=kudu/127.9.155.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:34135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:10:04.081378 11783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:04.081662 11783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:04.081710 11783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:04.085858 11783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:04.085919 11783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:04.085961 11783 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 19:10:04.085976 11783 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 19:10:04.086030 11783 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.66
I20260620 19:10:04.091575 11783 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:34135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.66.keytab
--principal=kudu/127.9.155.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.155.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.155.126:37761
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11783
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.66
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:10:04.092803 11783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:04.093869 11783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:10:04.104072 11791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:04.104084 11789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:04.104084 11788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:04.104399 11783 server_base.cc:1034] running on GCE node
I20260620 19:10:04.104892 11783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:04.105587 11783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:04.106782 11783 hybrid_clock.cc:648] HybridClock initialized: now 1781982604106757 us; error 38 us; skew 500 ppm
Jun 20 19:10:04 dist-test-slave-n3fx krb5kdc[11526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982604, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:04.109903 11783 init.cc:377] Logged in from keytab as kudu/127.9.155.66@KRBTEST.COM (short username kudu)
I20260620 19:10:04.110842 11783 webserver.cc:460] Webserver started at http://127.9.155.66:41289/ using document root <none> and password file <none>
I20260620 19:10:04.111421 11783 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:04.111495 11783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:04.111701 11783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:10:04.113826 11783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/minicluster-data/ts-1/data/instance:
uuid: "95a0516e6b0540bcb414fedc938f21bb"
format_stamp: "Formatted at 2026-06-20 19:10:04 on dist-test-slave-n3fx"
server_key: "2ce4025bc03be869246398decf4810ca"
server_key_iv: "69bada8fff7356673aa8f455ff85bc76"
server_key_version: "encryptionkey@0"
I20260620 19:10:04.114413 11783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/minicluster-data/ts-1/wal/instance:
uuid: "95a0516e6b0540bcb414fedc938f21bb"
format_stamp: "Formatted at 2026-06-20 19:10:04 on dist-test-slave-n3fx"
server_key: "2ce4025bc03be869246398decf4810ca"
server_key_iv: "69bada8fff7356673aa8f455ff85bc76"
server_key_version: "encryptionkey@0"
I20260620 19:10:04.118350 11783 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260620 19:10:04.120970 11798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:04.122344 11783 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:10:04.122468 11783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/minicluster-data/ts-1/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/minicluster-data/ts-1/wal
uuid: "95a0516e6b0540bcb414fedc938f21bb"
format_stamp: "Formatted at 2026-06-20 19:10:04 on dist-test-slave-n3fx"
server_key: "2ce4025bc03be869246398decf4810ca"
server_key_iv: "69bada8fff7356673aa8f455ff85bc76"
server_key_version: "encryptionkey@0"
I20260620 19:10:04.122581 11783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:04.142376 11783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:04.146050 11783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:04.146255 11783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:04.147832 11783 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:10:04.147892 11783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:04.148033 11783 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:10:04.148080 11783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 19:10:04 dist-test-slave-n3fx krb5kdc[11526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982604, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.66@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:10:04.158061 11783 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.66:40669
I20260620 19:10:04.158133 11913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.66:40669 every 8 connection(s)
I20260620 19:10:04.159139 11783 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
I20260620 19:10:04.161612 11620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:04.148522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.66:55741 (local address 127.9.155.126:37761)
0620 19:10:04.148680 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:04.148685 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:04.150074 (+ 1389us) server_negotiation.cc:407] Connection header received
0620 19:10:04.151337 (+ 1263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:04.151342 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:04.151399 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:04.151478 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:04.153231 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:04.153893 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:04.155085 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:04.155308 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:04.155395 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:04.158007 (+ 2612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:04.158011 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:04.158014 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:04.158044 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:04.159556 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:04.160168 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:04.160171 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:04.160172 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:04.160299 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:04.160577 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:04.160579 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:04.160580 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:04.160765 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:04.160840 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:04.161318 (+ 478us) server_negotiation.cc:299] Negotiation successful
0620 19:10:04.161490 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 19:10:04.162421 11807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:04.149380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:37761 (local address 127.9.155.66:55741)
0620 19:10:04.149877 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:04.149945 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:04.151079 (+ 1134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:04.151696 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:04.151706 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:04.152279 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:04.153051 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:04.153074 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:04.154043 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:04.154046 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:04.154963 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:04.154973 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:04.155458 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:04.155463 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:04.155622 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:04.156307 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:04.156329 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:04.157788 (+ 1459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:04.159713 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:04.159719 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:04.159735 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:04.160069 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:04.160374 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:04.160376 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:04.160378 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:04.160507 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:04.160875 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:04.160880 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:04.161121 (+ 241us) client_negotiation.cc:769] Sending connection context
0620 19:10:04.161289 (+ 168us) client_negotiation.cc:240] Negotiation successful
0620 19:10:04.161553 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":101,"threads_started":1}
I20260620 19:10:04.163663 11914 heartbeater.cc:347] Connected to a master server at 127.9.155.126:37761
I20260620 19:10:04.163817 11914 heartbeater.cc:464] Registering TS with master...
I20260620 19:10:04.164419 11914 heartbeater.cc:511] Master 127.9.155.126:37761 requested a full tablet report, sending...
I20260620 19:10:04.165567 11570 ts_manager.cc:194] Registered new tserver with Master: 95a0516e6b0540bcb414fedc938f21bb (127.9.155.66:40669)
I20260620 19:10:04.166350 11570 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.155.66@KRBTEST.COM'} at 127.9.155.66:55741
I20260620 19:10:04.167802 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 11783
I20260620 19:10:04.167907 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/minicluster-data/ts-1/wal/instance
I20260620 19:10:04.168192 9837 external_mini_cluster.cc:1428] Setting key 06ce2871ea11c2430e49b2f4e5623ae0
I20260620 19:10:04.173794 11920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:04.165968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:37761 (local address 127.9.155.66:58945)
0620 19:10:04.166339 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:04.166353 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:04.166457 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:04.166716 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:04.166719 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:04.166950 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:04.167115 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:04.167122 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:04.168148 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:04.168152 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:04.168955 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:04.168964 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:04.169499 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:04.169502 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:04.169585 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:04.170058 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:04.170072 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:04.170473 (+ 401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:04.172085 (+ 1612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:04.172088 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:04.172091 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:04.172346 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:04.172628 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:04.172632 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:04.172634 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:04.172786 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:04.173304 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:04.173307 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:04.173395 (+ 88us) client_negotiation.cc:769] Sending connection context
0620 19:10:04.173490 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 19:10:04.173621 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":103,"threads_started":1}
I20260620 19:10:04.174014 11620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:04.166072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.66:58945 (local address 127.9.155.126:37761)
0620 19:10:04.166178 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:04.166195 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:04.166441 (+ 246us) server_negotiation.cc:407] Connection header received
0620 19:10:04.166550 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:04.166553 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:04.166616 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:04.166688 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:04.167250 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:04.168018 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:04.169115 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:04.169398 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:04.169497 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:04.170602 (+ 1105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:04.170605 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:04.170607 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:04.170629 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:04.171965 (+ 1336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:04.172433 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:04.172437 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:04.172439 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:04.172525 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:04.172890 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:04.172893 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:04.172894 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:04.173191 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:04.173286 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:04.173560 (+ 274us) server_negotiation.cc:299] Negotiation successful
0620 19:10:04.173685 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
WARNING: no policy specified for kudu/127.9.155.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.155.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.155.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.67.keytab.
Entry for principal kudu/127.9.155.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.67.keytab.
I20260620 19:10:04.195569 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.67:0
--local_ip_for_outbound_sockets=127.9.155.67
--webserver_interface=127.9.155.67
--webserver_port=0
--tserver_master_addrs=127.9.155.126:37761
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.67.keytab
--principal=kudu/127.9.155.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:34135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:10:04.307199 11923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:04.307464 11923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:04.307533 11923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:04.311655 11923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:04.311712 11923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:04.311769 11923 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 19:10:04.311793 11923 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 19:10:04.311842 11923 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.67
I20260620 19:10:04.317150 11923 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:34135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.67.keytab
--principal=kudu/127.9.155.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.155.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.155.126:37761
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11923
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.67
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:10:04.318184 11923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:04.319391 11923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 19:10:04.330041 11923 server_base.cc:1034] running on GCE node
W20260620 19:10:04.330015 11929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:04.330049 11928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:04.330060 11931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:04.330698 11923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:04.331435 11923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:04.332597 11923 hybrid_clock.cc:648] HybridClock initialized: now 1781982604332576 us; error 35 us; skew 500 ppm
Jun 20 19:10:04 dist-test-slave-n3fx krb5kdc[11526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982604, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:04.335359 11923 init.cc:377] Logged in from keytab as kudu/127.9.155.67@KRBTEST.COM (short username kudu)
I20260620 19:10:04.336438 11923 webserver.cc:460] Webserver started at http://127.9.155.67:35149/ using document root <none> and password file <none>
I20260620 19:10:04.336989 11923 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:04.337047 11923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:04.337217 11923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:10:04.339274 11923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/minicluster-data/ts-2/data/instance:
uuid: "694679b53ba0487495655430a0617582"
format_stamp: "Formatted at 2026-06-20 19:10:04 on dist-test-slave-n3fx"
server_key: "6354f7418e3735277bc75ada68fabf37"
server_key_iv: "9a6947f775e08229da30802a8e2e7af7"
server_key_version: "encryptionkey@0"
I20260620 19:10:04.339851 11923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/minicluster-data/ts-2/wal/instance:
uuid: "694679b53ba0487495655430a0617582"
format_stamp: "Formatted at 2026-06-20 19:10:04 on dist-test-slave-n3fx"
server_key: "6354f7418e3735277bc75ada68fabf37"
server_key_iv: "9a6947f775e08229da30802a8e2e7af7"
server_key_version: "encryptionkey@0"
I20260620 19:10:04.344069 11923 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260620 19:10:04.346751 11938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:04.348057 11923 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 19:10:04.348157 11923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/minicluster-data/ts-2/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/minicluster-data/ts-2/wal
uuid: "694679b53ba0487495655430a0617582"
format_stamp: "Formatted at 2026-06-20 19:10:04 on dist-test-slave-n3fx"
server_key: "6354f7418e3735277bc75ada68fabf37"
server_key_iv: "9a6947f775e08229da30802a8e2e7af7"
server_key_version: "encryptionkey@0"
I20260620 19:10:04.348295 11923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:04.370771 11923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:04.373921 11923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:04.374096 11923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:04.375674 11923 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:10:04.375725 11923 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:04.375797 11923 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:10:04.375823 11923 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 19:10:04 dist-test-slave-n3fx krb5kdc[11526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982604, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.67@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:10:04.385744 11923 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.67:34297
I20260620 19:10:04.386031 12053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.67:34297 every 8 connection(s)
I20260620 19:10:04.386710 11923 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
I20260620 19:10:04.388538 11620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:04.376508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.67:43925 (local address 127.9.155.126:37761)
0620 19:10:04.376647 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:04.376652 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:04.377491 (+ 839us) server_negotiation.cc:407] Connection header received
0620 19:10:04.378504 (+ 1013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:04.378508 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:04.378563 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:04.378641 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:04.379800 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:04.380414 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:04.381644 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:04.381929 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:04.382032 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:04.384905 (+ 2873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:04.384908 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:04.384910 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:04.384936 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:04.386442 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:04.386973 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:04.386976 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:04.386977 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:04.387059 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:04.387463 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:04.387466 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:04.387468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:04.387689 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:04.387768 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:04.388311 (+ 543us) server_negotiation.cc:299] Negotiation successful
0620 19:10:04.388418 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 19:10:04.389240 11947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:04.376914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:37761 (local address 127.9.155.67:43925)
0620 19:10:04.377336 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:04.377384 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:04.378254 (+ 870us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:04.378838 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:04.378845 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:04.379156 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:04.379660 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:04.379676 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:04.380537 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:04.380540 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:04.381525 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:04.381535 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:04.382072 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:04.382076 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:04.382243 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:04.382854 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:04.382874 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:04.384689 (+ 1815us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:04.386546 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:04.386552 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:04.386565 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:04.386887 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:04.387195 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:04.387198 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:04.387200 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:04.387361 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:04.387789 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:04.387795 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:04.388044 (+ 249us) client_negotiation.cc:769] Sending connection context
0620 19:10:04.388199 (+ 155us) client_negotiation.cc:240] Negotiation successful
0620 19:10:04.388494 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":95,"threads_started":1}
I20260620 19:10:04.390323 12054 heartbeater.cc:347] Connected to a master server at 127.9.155.126:37761
I20260620 19:10:04.390455 12054 heartbeater.cc:464] Registering TS with master...
I20260620 19:10:04.390913 12054 heartbeater.cc:511] Master 127.9.155.126:37761 requested a full tablet report, sending...
I20260620 19:10:04.392036 11571 ts_manager.cc:194] Registered new tserver with Master: 694679b53ba0487495655430a0617582 (127.9.155.67:34297)
I20260620 19:10:04.392190 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 11923
I20260620 19:10:04.392339 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/minicluster-data/ts-2/wal/instance
I20260620 19:10:04.392624 9837 external_mini_cluster.cc:1428] Setting key 497edd6ba41d1f0d51ed70f042d0951d
I20260620 19:10:04.392791 11571 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.155.67@KRBTEST.COM'} at 127.9.155.67:43925
I20260620 19:10:04.394598 9837 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:10:04.400437 11620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:04.392985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.67:45323 (local address 127.9.155.126:37761)
0620 19:10:04.393088 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:04.393092 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:04.393103 (+ 11us) server_negotiation.cc:407] Connection header received
0620 19:10:04.393134 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:04.393137 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:04.393178 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:04.393255 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:04.393834 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:04.394367 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:04.395685 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:04.395915 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:04.396006 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:04.397118 (+ 1112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:04.397122 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:04.397125 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:04.397152 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:04.398643 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:04.399107 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:04.399110 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:04.399112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:04.399197 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:04.399519 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:04.399522 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:04.399523 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:04.399734 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:04.399829 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:04.400136 (+ 307us) server_negotiation.cc:299] Negotiation successful
0620 19:10:04.400259 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 19:10:04.400473 12060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:04.392500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:37761 (local address 127.9.155.67:45323)
0620 19:10:04.392804 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:04.392820 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:04.392910 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:04.393358 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:04.393361 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:04.393589 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:04.393701 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:04.393709 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:04.394508 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:04.394511 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:04.395548 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:04.395559 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:04.396022 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:04.396025 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:04.396117 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:04.396669 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:04.396679 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:04.396985 (+ 306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:04.398764 (+ 1779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:04.398769 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:04.398771 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:04.399014 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:04.399333 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:04.399336 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:04.399338 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:04.399424 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:04.399848 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:04.399851 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:04.399953 (+ 102us) client_negotiation.cc:769] Sending connection context
0620 19:10:04.400090 (+ 137us) client_negotiation.cc:240] Negotiation successful
0620 19:10:04.400286 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":84,"threads_started":1}
Jun 20 19:10:04 dist-test-slave-n3fx krb5kdc[11526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982604, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 19:10:04 dist-test-slave-n3fx krb5kdc[11526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982604, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:10:04.420941 11620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:04.412298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53092 (local address 127.9.155.126:37761)
0620 19:10:04.412419 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:04.412423 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:04.412435 (+ 12us) server_negotiation.cc:407] Connection header received
0620 19:10:04.412474 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:04.412477 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:04.412515 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:04.412581 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:04.413289 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:04.413825 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:04.414701 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:04.414938 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:04.415031 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:04.417636 (+ 2605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:04.417639 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:04.417641 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:04.417676 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:04.419021 (+ 1345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:04.419444 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:04.419447 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:04.419448 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:04.419541 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:04.419866 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:04.419869 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:04.419870 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:04.420095 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:04.420198 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:04.420660 (+ 462us) server_negotiation.cc:299] Negotiation successful
0620 19:10:04.420813 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 19:10:04.424322 11571 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53092:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:10:04.427577 11571 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:10:04.442304 12075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:04.436561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.67:34297 (local address 127.0.0.1:44422)
0620 19:10:04.437569 (+ 1008us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:04.437582 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:04.437668 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:04.438225 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:04.438228 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:04.438244 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:10:04.438325 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:04.438330 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:04.439650 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:04.439653 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:04.440675 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:04.440684 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:04.441846 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:04.441848 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:04.441909 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:04.441963 (+ 54us) client_negotiation.cc:769] Sending connection context
0620 19:10:04.442054 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 19:10:04.442144 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":949,"mutex_wait_us":35,"thread_start_us":78,"threads_started":1}
I20260620 19:10:04.442341 12076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:04.437041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44422 (local address 127.9.155.67:34297)
0620 19:10:04.437501 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:04.437511 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:04.437666 (+ 155us) server_negotiation.cc:407] Connection header received
0620 19:10:04.437765 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:04.437770 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:04.437935 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:04.438083 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:10:04.438448 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:04.439434 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:04.440785 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:04.441589 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:04.441698 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:04.441864 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:04.442099 (+ 235us) server_negotiation.cc:299] Negotiation successful
0620 19:10:04.442196 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":111,"threads_started":1}
I20260620 19:10:04.443368 12071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:04.435755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.65:43623 (local address 127.0.0.1:40316)
0620 19:10:04.436842 (+ 1087us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:04.436889 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:04.437101 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:04.437828 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:04.437833 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:04.437850 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:10:04.437959 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:04.437963 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:04.439186 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:04.439189 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:04.440724 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:04.440736 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:04.442967 (+ 2231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:04.442971 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:04.443060 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:04.443083 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 19:10:04.443150 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 19:10:04.443222 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":849,"mutex_wait_us":134,"spinlock_wait_cycles":9728,"thread_start_us":107,"threads_started":1}
I20260620 19:10:04.443397 12074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:04.436157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40316 (local address 127.9.155.65:43623)
0620 19:10:04.436513 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:04.436528 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:04.436987 (+ 459us) server_negotiation.cc:407] Connection header received
0620 19:10:04.437288 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:04.437294 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:04.437513 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:04.437640 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:10:04.438064 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:04.439080 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:04.441923 (+ 2843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:04.442673 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:04.442772 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:04.442905 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:04.443191 (+ 286us) server_negotiation.cc:299] Negotiation successful
0620 19:10:04.443282 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":114,"threads_started":1}
I20260620 19:10:04.444664 12072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:04.435724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.66:40669 (local address 127.0.0.1:47168)
0620 19:10:04.436761 (+ 1037us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:04.436895 (+ 134us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:04.437102 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:04.437839 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:04.437842 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:04.437858 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:10:04.437980 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:04.437987 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:04.439065 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:04.439069 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:04.440733 (+ 1664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:04.440738 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:04.441707 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:04.441711 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:04.442499 (+ 788us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:04.442522 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 19:10:04.444424 (+ 1902us) client_negotiation.cc:240] Negotiation successful
0620 19:10:04.444529 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":864,"mutex_wait_us":146,"spinlock_wait_cycles":3840,"thread_start_us":105,"threads_started":1}
I20260620 19:10:04.444727 12073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:04.436107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47168 (local address 127.9.155.66:40669)
0620 19:10:04.436536 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:04.436546 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:04.436989 (+ 443us) server_negotiation.cc:407] Connection header received
0620 19:10:04.437294 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:04.437300 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:04.437514 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:04.437635 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:10:04.438106 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:04.438960 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:04.440826 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:04.441589 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:04.441698 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:04.441864 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:04.444476 (+ 2612us) server_negotiation.cc:299] Negotiation successful
0620 19:10:04.444583 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":120,"threads_started":1}
I20260620 19:10:04.445562 11988 tablet_service.cc:1467] Processing CreateTablet for tablet 293e3d9f325a4239822a43e01f439391 (DEFAULT_TABLE table=test-table [id=e4b70625f7eb4d64b515906c87c719cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:10:04.446064 11708 tablet_service.cc:1467] Processing CreateTablet for tablet 293e3d9f325a4239822a43e01f439391 (DEFAULT_TABLE table=test-table [id=e4b70625f7eb4d64b515906c87c719cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:10:04.446537 11988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 293e3d9f325a4239822a43e01f439391. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:04.446915 11708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 293e3d9f325a4239822a43e01f439391. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:04.452970 12077 tablet_bootstrap.cc:492] T 293e3d9f325a4239822a43e01f439391 P be1d9954fe514ae18445ac0250dae2d8: Bootstrap starting.
I20260620 19:10:04.454181 12078 tablet_bootstrap.cc:492] T 293e3d9f325a4239822a43e01f439391 P 694679b53ba0487495655430a0617582: Bootstrap starting.
I20260620 19:10:04.454459 11848 tablet_service.cc:1467] Processing CreateTablet for tablet 293e3d9f325a4239822a43e01f439391 (DEFAULT_TABLE table=test-table [id=e4b70625f7eb4d64b515906c87c719cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:10:04.455552 11848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 293e3d9f325a4239822a43e01f439391. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:04.455736 12077 tablet_bootstrap.cc:654] T 293e3d9f325a4239822a43e01f439391 P be1d9954fe514ae18445ac0250dae2d8: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:04.456262 12078 tablet_bootstrap.cc:654] T 293e3d9f325a4239822a43e01f439391 P 694679b53ba0487495655430a0617582: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:04.456468 12077 log.cc:824] T 293e3d9f325a4239822a43e01f439391 P be1d9954fe514ae18445ac0250dae2d8: Log is configured to *not* fsync() on all Append() calls
I20260620 19:10:04.457046 12078 log.cc:824] T 293e3d9f325a4239822a43e01f439391 P 694679b53ba0487495655430a0617582: Log is configured to *not* fsync() on all Append() calls
I20260620 19:10:04.458559 12077 tablet_bootstrap.cc:492] T 293e3d9f325a4239822a43e01f439391 P be1d9954fe514ae18445ac0250dae2d8: No bootstrap required, opened a new log
I20260620 19:10:04.458752 12077 ts_tablet_manager.cc:1397] T 293e3d9f325a4239822a43e01f439391 P be1d9954fe514ae18445ac0250dae2d8: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 19:10:04.459088 12078 tablet_bootstrap.cc:492] T 293e3d9f325a4239822a43e01f439391 P 694679b53ba0487495655430a0617582: No bootstrap required, opened a new log
I20260620 19:10:04.459285 12078 ts_tablet_manager.cc:1397] T 293e3d9f325a4239822a43e01f439391 P 694679b53ba0487495655430a0617582: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260620 19:10:04.461642 12077 raft_consensus.cc:348] T 293e3d9f325a4239822a43e01f439391 P be1d9954fe514ae18445ac0250dae2d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a0516e6b0540bcb414fedc938f21bb" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40669 } } peers { permanent_uuid: "be1d9954fe514ae18445ac0250dae2d8" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 43623 } } peers { permanent_uuid: "694679b53ba0487495655430a0617582" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34297 } }
I20260620 19:10:04.461870 12077 raft_consensus.cc:374] T 293e3d9f325a4239822a43e01f439391 P be1d9954fe514ae18445ac0250dae2d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:04.461922 12077 raft_consensus.cc:729] T 293e3d9f325a4239822a43e01f439391 P be1d9954fe514ae18445ac0250dae2d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be1d9954fe514ae18445ac0250dae2d8, State: Initialized, Role: FOLLOWER
I20260620 19:10:04.462323 12077 consensus_queue.cc:260] T 293e3d9f325a4239822a43e01f439391 P be1d9954fe514ae18445ac0250dae2d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a0516e6b0540bcb414fedc938f21bb" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40669 } } peers { permanent_uuid: "be1d9954fe514ae18445ac0250dae2d8" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 43623 } } peers { permanent_uuid: "694679b53ba0487495655430a0617582" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34297 } }
I20260620 19:10:04.463104 12077 ts_tablet_manager.cc:1428] T 293e3d9f325a4239822a43e01f439391 P be1d9954fe514ae18445ac0250dae2d8: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260620 19:10:04.462977 12078 raft_consensus.cc:348] T 293e3d9f325a4239822a43e01f439391 P 694679b53ba0487495655430a0617582 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a0516e6b0540bcb414fedc938f21bb" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40669 } } peers { permanent_uuid: "be1d9954fe514ae18445ac0250dae2d8" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 43623 } } peers { permanent_uuid: "694679b53ba0487495655430a0617582" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34297 } }
I20260620 19:10:04.463265 12078 raft_consensus.cc:374] T 293e3d9f325a4239822a43e01f439391 P 694679b53ba0487495655430a0617582 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:04.463322 12078 raft_consensus.cc:729] T 293e3d9f325a4239822a43e01f439391 P 694679b53ba0487495655430a0617582 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 694679b53ba0487495655430a0617582, State: Initialized, Role: FOLLOWER
I20260620 19:10:04.463460 11774 heartbeater.cc:503] Master 127.9.155.126:37761 was elected leader, sending a full tablet report...
I20260620 19:10:04.463670 12082 tablet_bootstrap.cc:492] T 293e3d9f325a4239822a43e01f439391 P 95a0516e6b0540bcb414fedc938f21bb: Bootstrap starting.
I20260620 19:10:04.463759 12078 consensus_queue.cc:260] T 293e3d9f325a4239822a43e01f439391 P 694679b53ba0487495655430a0617582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a0516e6b0540bcb414fedc938f21bb" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40669 } } peers { permanent_uuid: "be1d9954fe514ae18445ac0250dae2d8" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 43623 } } peers { permanent_uuid: "694679b53ba0487495655430a0617582" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34297 } }
I20260620 19:10:04.464442 12054 heartbeater.cc:503] Master 127.9.155.126:37761 was elected leader, sending a full tablet report...
I20260620 19:10:04.464783 12078 ts_tablet_manager.cc:1428] T 293e3d9f325a4239822a43e01f439391 P 694679b53ba0487495655430a0617582: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260620 19:10:04.466379 12082 tablet_bootstrap.cc:654] T 293e3d9f325a4239822a43e01f439391 P 95a0516e6b0540bcb414fedc938f21bb: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:04.467151 12082 log.cc:824] T 293e3d9f325a4239822a43e01f439391 P 95a0516e6b0540bcb414fedc938f21bb: Log is configured to *not* fsync() on all Append() calls
I20260620 19:10:04.468911 12082 tablet_bootstrap.cc:492] T 293e3d9f325a4239822a43e01f439391 P 95a0516e6b0540bcb414fedc938f21bb: No bootstrap required, opened a new log
I20260620 19:10:04.469075 12082 ts_tablet_manager.cc:1397] T 293e3d9f325a4239822a43e01f439391 P 95a0516e6b0540bcb414fedc938f21bb: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 19:10:04.471783 12082 raft_consensus.cc:348] T 293e3d9f325a4239822a43e01f439391 P 95a0516e6b0540bcb414fedc938f21bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a0516e6b0540bcb414fedc938f21bb" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40669 } } peers { permanent_uuid: "be1d9954fe514ae18445ac0250dae2d8" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 43623 } } peers { permanent_uuid: "694679b53ba0487495655430a0617582" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34297 } }
I20260620 19:10:04.471995 12082 raft_consensus.cc:374] T 293e3d9f325a4239822a43e01f439391 P 95a0516e6b0540bcb414fedc938f21bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:04.472026 12082 raft_consensus.cc:729] T 293e3d9f325a4239822a43e01f439391 P 95a0516e6b0540bcb414fedc938f21bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95a0516e6b0540bcb414fedc938f21bb, State: Initialized, Role: FOLLOWER
I20260620 19:10:04.472420 12082 consensus_queue.cc:260] T 293e3d9f325a4239822a43e01f439391 P 95a0516e6b0540bcb414fedc938f21bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a0516e6b0540bcb414fedc938f21bb" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40669 } } peers { permanent_uuid: "be1d9954fe514ae18445ac0250dae2d8" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 43623 } } peers { permanent_uuid: "694679b53ba0487495655430a0617582" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34297 } }
I20260620 19:10:04.473052 11914 heartbeater.cc:503] Master 127.9.155.126:37761 was elected leader, sending a full tablet report...
I20260620 19:10:04.473345 12082 ts_tablet_manager.cc:1428] T 293e3d9f325a4239822a43e01f439391 P 95a0516e6b0540bcb414fedc938f21bb: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20260620 19:10:04.637691 12055 tablet.cc:2369] T 293e3d9f325a4239822a43e01f439391 P 694679b53ba0487495655430a0617582: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 19:10:04.660470 11915 tablet.cc:2369] T 293e3d9f325a4239822a43e01f439391 P 95a0516e6b0540bcb414fedc938f21bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 19:10:04.687683 11775 tablet.cc:2369] T 293e3d9f325a4239822a43e01f439391 P be1d9954fe514ae18445ac0250dae2d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:10:04.707546 12083 raft_consensus.cc:482] T 293e3d9f325a4239822a43e01f439391 P 694679b53ba0487495655430a0617582 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:10:04.707763 12083 raft_consensus.cc:504] T 293e3d9f325a4239822a43e01f439391 P 694679b53ba0487495655430a0617582 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a0516e6b0540bcb414fedc938f21bb" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40669 } } peers { permanent_uuid: "be1d9954fe514ae18445ac0250dae2d8" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 43623 } } peers { permanent_uuid: "694679b53ba0487495655430a0617582" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34297 } }
I20260620 19:10:04.709008 12083 leader_election.cc:288] T 293e3d9f325a4239822a43e01f439391 P 694679b53ba0487495655430a0617582 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 95a0516e6b0540bcb414fedc938f21bb (127.9.155.66:40669), be1d9954fe514ae18445ac0250dae2d8 (127.9.155.65:43623)
I20260620 19:10:04.713245 12073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:04.709372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.67:36707 (local address 127.9.155.66:40669)
0620 19:10:04.709494 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:04.709498 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:04.709515 (+ 17us) server_negotiation.cc:407] Connection header received
0620 19:10:04.709645 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:04.709648 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:04.709700 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:04.709777 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:10:04.710058 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:04.710684 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:04.712084 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:04.712754 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:04.712877 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:04.712932 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:04.713071 (+ 139us) server_negotiation.cc:299] Negotiation successful
0620 19:10:04.713139 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 19:10:04.713246 11947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:04.709280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.66:40669 (local address 127.9.155.67:36707)
0620 19:10:04.709405 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:04.709421 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:04.709517 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:04.709807 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:04.709810 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:04.709830 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:10:04.709948 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:04.709955 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:04.710808 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:04.710810 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:04.711957 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:04.711966 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:04.712864 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:04.712867 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:04.712948 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:04.712970 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 19:10:04.713041 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 19:10:04.713112 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260620 19:10:04.714057 12086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:04.709282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.65:43623 (local address 127.9.155.67:53137)
0620 19:10:04.709841 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:04.709854 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:04.709956 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:04.710205 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:04.710207 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:04.710234 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:10:04.710789 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:04.710800 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:04.711787 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:04.711790 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:04.712901 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:04.712910 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:04.713715 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:04.713719 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:04.713816 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:04.713839 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 19:10:04.713900 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 19:10:04.713948 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"mutex_wait_us":211,"thread_start_us":111,"threads_started":1}
I20260620 19:10:04.714116 12074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:04.709355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.67:53137 (local address 127.9.155.65:43623)
0620 19:10:04.709480 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:04.709484 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:04.709937 (+ 453us) server_negotiation.cc:407] Connection header received
0620 19:10:04.710065 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:04.710069 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:04.710115 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:04.710194 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:10:04.710897 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:04.711654 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:04.713028 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:04.713584 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:04.713684 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:04.713718 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:04.713937 (+ 219us) server_negotiation.cc:299] Negotiation successful
0620 19:10:04.714010 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 19:10:04.714077 11868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293e3d9f325a4239822a43e01f439391" candidate_uuid: "694679b53ba0487495655430a0617582" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95a0516e6b0540bcb414fedc938f21bb" is_pre_election: true
I20260620 19:10:04.714401 11868 raft_consensus.cc:2454] T 293e3d9f325a4239822a43e01f439391 P 95a0516e6b0540bcb414fedc938f21bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 694679b53ba0487495655430a0617582 in term 0.
I20260620 19:10:04.714709 11719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293e3d9f325a4239822a43e01f439391" candidate_uuid: "694679b53ba0487495655430a0617582" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be1d9954fe514ae18445ac0250dae2d8" is_pre_election: true
I20260620 19:10:04.714913 11939 leader_election.cc:302] T 293e3d9f325a4239822a43e01f439391 P 694679b53ba0487495655430a0617582 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 694679b53ba0487495655430a0617582, 95a0516e6b0540bcb414fedc938f21bb; no voters:
I20260620 19:10:04.714986 11719 raft_consensus.cc:2454] T 293e3d9f325a4239822a43e01f439391 P be1d9954fe514ae18445ac0250dae2d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 694679b53ba0487495655430a0617582 in term 0.
I20260620 19:10:04.715188 12083 raft_consensus.cc:2781] T 293e3d9f325a4239822a43e01f439391 P 694679b53ba0487495655430a0617582 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:10:04.715291 12083 raft_consensus.cc:482] T 293e3d9f325a4239822a43e01f439391 P 694679b53ba0487495655430a0617582 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:10:04.715348 12083 raft_consensus.cc:3037] T 293e3d9f325a4239822a43e01f439391 P 694679b53ba0487495655430a0617582 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:04.716398 12083 raft_consensus.cc:504] T 293e3d9f325a4239822a43e01f439391 P 694679b53ba0487495655430a0617582 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a0516e6b0540bcb414fedc938f21bb" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40669 } } peers { permanent_uuid: "be1d9954fe514ae18445ac0250dae2d8" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 43623 } } peers { permanent_uuid: "694679b53ba0487495655430a0617582" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34297 } }
I20260620 19:10:04.716701 12083 leader_election.cc:288] T 293e3d9f325a4239822a43e01f439391 P 694679b53ba0487495655430a0617582 [CANDIDATE]: Term 1 election: Requested vote from peers 95a0516e6b0540bcb414fedc938f21bb (127.9.155.66:40669), be1d9954fe514ae18445ac0250dae2d8 (127.9.155.65:43623)
I20260620 19:10:04.717041 11719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293e3d9f325a4239822a43e01f439391" candidate_uuid: "694679b53ba0487495655430a0617582" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be1d9954fe514ae18445ac0250dae2d8"
I20260620 19:10:04.717051 11868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293e3d9f325a4239822a43e01f439391" candidate_uuid: "694679b53ba0487495655430a0617582" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95a0516e6b0540bcb414fedc938f21bb"
I20260620 19:10:04.717173 11719 raft_consensus.cc:3037] T 293e3d9f325a4239822a43e01f439391 P be1d9954fe514ae18445ac0250dae2d8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:04.717195 11868 raft_consensus.cc:3037] T 293e3d9f325a4239822a43e01f439391 P 95a0516e6b0540bcb414fedc938f21bb [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:04.718416 11868 raft_consensus.cc:2454] T 293e3d9f325a4239822a43e01f439391 P 95a0516e6b0540bcb414fedc938f21bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 694679b53ba0487495655430a0617582 in term 1.
I20260620 19:10:04.718416 11719 raft_consensus.cc:2454] T 293e3d9f325a4239822a43e01f439391 P be1d9954fe514ae18445ac0250dae2d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 694679b53ba0487495655430a0617582 in term 1.
I20260620 19:10:04.718765 11939 leader_election.cc:302] T 293e3d9f325a4239822a43e01f439391 P 694679b53ba0487495655430a0617582 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 694679b53ba0487495655430a0617582, 95a0516e6b0540bcb414fedc938f21bb; no voters:
I20260620 19:10:04.718966 12083 raft_consensus.cc:2781] T 293e3d9f325a4239822a43e01f439391 P 694679b53ba0487495655430a0617582 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:10:04.719200 12083 raft_consensus.cc:686] T 293e3d9f325a4239822a43e01f439391 P 694679b53ba0487495655430a0617582 [term 1 LEADER]: Becoming Leader. State: Replica: 694679b53ba0487495655430a0617582, State: Running, Role: LEADER
I20260620 19:10:04.719513 12083 consensus_queue.cc:237] T 293e3d9f325a4239822a43e01f439391 P 694679b53ba0487495655430a0617582 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a0516e6b0540bcb414fedc938f21bb" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40669 } } peers { permanent_uuid: "be1d9954fe514ae18445ac0250dae2d8" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 43623 } } peers { permanent_uuid: "694679b53ba0487495655430a0617582" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34297 } }
I20260620 19:10:04.722877 11570 catalog_manager.cc:5495] T 293e3d9f325a4239822a43e01f439391 P 694679b53ba0487495655430a0617582 reported cstate change: term changed from 0 to 1, leader changed from <none> to 694679b53ba0487495655430a0617582 (127.9.155.67). New cstate: current_term: 1 leader_uuid: "694679b53ba0487495655430a0617582" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95a0516e6b0540bcb414fedc938f21bb" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40669 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be1d9954fe514ae18445ac0250dae2d8" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 43623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "694679b53ba0487495655430a0617582" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34297 } health_report { overall_health: HEALTHY } } }
Jun 20 19:10:04 dist-test-slave-n3fx krb5kdc[11526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982603, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.126@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:10:04.758159 12094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:04.749979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:37761 (local address 127.0.0.1:53102)
0620 19:10:04.750256 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:04.750269 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:04.750408 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:04.750634 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:04.750636 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:04.751021 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:04.751137 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:04.751144 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:04.752015 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:04.752017 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:04.752715 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:04.752725 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:04.753177 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:04.753180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:04.753245 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:04.753836 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:04.753848 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:04.755271 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:04.756803 (+ 1532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:04.756812 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:04.756817 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:04.757036 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:04.757313 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:04.757316 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:04.757317 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:04.757391 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:04.757693 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:04.757699 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:04.757823 (+ 124us) client_negotiation.cc:769] Sending connection context
0620 19:10:04.757914 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 19:10:04.758023 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":102,"threads_started":1}
I20260620 19:10:04.758195 11620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:04.750040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53102 (local address 127.9.155.126:37761)
0620 19:10:04.750171 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:04.750175 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:04.750366 (+ 191us) server_negotiation.cc:407] Connection header received
0620 19:10:04.750486 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:04.750489 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:04.750533 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:04.750608 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:04.751254 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:04.751875 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:04.752843 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:04.753067 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:04.753151 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:04.755411 (+ 2260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:04.755414 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:04.755416 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:04.755437 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:04.756696 (+ 1259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:04.757115 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:04.757117 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:04.757118 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:04.757191 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:04.757454 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:04.757456 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:04.757457 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:04.757619 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:04.757686 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:04.757993 (+ 307us) server_negotiation.cc:299] Negotiation successful
0620 19:10:04.758087 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 19:10:04.762043 11570 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.9.155.126@KRBTEST.COM'} at 127.0.0.1:53102:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 19:10:04.762776 11570 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:10:04.767421 11708 tablet_service.cc:1467] Processing CreateTablet for tablet 68f1ae8958a74a2ebe17f3f4e5346e24 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=faf608141e7648a2a5a60a7ca0f52a8b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 19:10:04.767421 11848 tablet_service.cc:1467] Processing CreateTablet for tablet 68f1ae8958a74a2ebe17f3f4e5346e24 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=faf608141e7648a2a5a60a7ca0f52a8b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 19:10:04.767652 11988 tablet_service.cc:1467] Processing CreateTablet for tablet 68f1ae8958a74a2ebe17f3f4e5346e24 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=faf608141e7648a2a5a60a7ca0f52a8b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 19:10:04.768193 11708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68f1ae8958a74a2ebe17f3f4e5346e24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:04.768205 11848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68f1ae8958a74a2ebe17f3f4e5346e24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:04.768313 11988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68f1ae8958a74a2ebe17f3f4e5346e24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:04.770663 12078 tablet_bootstrap.cc:492] T 68f1ae8958a74a2ebe17f3f4e5346e24 P 694679b53ba0487495655430a0617582: Bootstrap starting.
I20260620 19:10:04.771067 12077 tablet_bootstrap.cc:492] T 68f1ae8958a74a2ebe17f3f4e5346e24 P be1d9954fe514ae18445ac0250dae2d8: Bootstrap starting.
I20260620 19:10:04.771067 12082 tablet_bootstrap.cc:492] T 68f1ae8958a74a2ebe17f3f4e5346e24 P 95a0516e6b0540bcb414fedc938f21bb: Bootstrap starting.
I20260620 19:10:04.771867 12078 tablet_bootstrap.cc:654] T 68f1ae8958a74a2ebe17f3f4e5346e24 P 694679b53ba0487495655430a0617582: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:04.772032 12077 tablet_bootstrap.cc:654] T 68f1ae8958a74a2ebe17f3f4e5346e24 P be1d9954fe514ae18445ac0250dae2d8: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:04.772032 12082 tablet_bootstrap.cc:654] T 68f1ae8958a74a2ebe17f3f4e5346e24 P 95a0516e6b0540bcb414fedc938f21bb: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:04.773252 12078 tablet_bootstrap.cc:492] T 68f1ae8958a74a2ebe17f3f4e5346e24 P 694679b53ba0487495655430a0617582: No bootstrap required, opened a new log
I20260620 19:10:04.773247 12082 tablet_bootstrap.cc:492] T 68f1ae8958a74a2ebe17f3f4e5346e24 P 95a0516e6b0540bcb414fedc938f21bb: No bootstrap required, opened a new log
I20260620 19:10:04.773355 12082 ts_tablet_manager.cc:1397] T 68f1ae8958a74a2ebe17f3f4e5346e24 P 95a0516e6b0540bcb414fedc938f21bb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:10:04.773335 12078 ts_tablet_manager.cc:1397] T 68f1ae8958a74a2ebe17f3f4e5346e24 P 694679b53ba0487495655430a0617582: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 19:10:04.773579 12077 tablet_bootstrap.cc:492] T 68f1ae8958a74a2ebe17f3f4e5346e24 P be1d9954fe514ae18445ac0250dae2d8: No bootstrap required, opened a new log
I20260620 19:10:04.773656 12077 ts_tablet_manager.cc:1397] T 68f1ae8958a74a2ebe17f3f4e5346e24 P be1d9954fe514ae18445ac0250dae2d8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 19:10:04.773788 12078 raft_consensus.cc:348] T 68f1ae8958a74a2ebe17f3f4e5346e24 P 694679b53ba0487495655430a0617582 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1d9954fe514ae18445ac0250dae2d8" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 43623 } } peers { permanent_uuid: "95a0516e6b0540bcb414fedc938f21bb" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40669 } } peers { permanent_uuid: "694679b53ba0487495655430a0617582" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34297 } }
I20260620 19:10:04.773788 12082 raft_consensus.cc:348] T 68f1ae8958a74a2ebe17f3f4e5346e24 P 95a0516e6b0540bcb414fedc938f21bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1d9954fe514ae18445ac0250dae2d8" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 43623 } } peers { permanent_uuid: "95a0516e6b0540bcb414fedc938f21bb" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40669 } } peers { permanent_uuid: "694679b53ba0487495655430a0617582" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34297 } }
I20260620 19:10:04.773911 12082 raft_consensus.cc:374] T 68f1ae8958a74a2ebe17f3f4e5346e24 P 95a0516e6b0540bcb414fedc938f21bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:04.773911 12078 raft_consensus.cc:374] T 68f1ae8958a74a2ebe17f3f4e5346e24 P 694679b53ba0487495655430a0617582 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:04.773949 12082 raft_consensus.cc:729] T 68f1ae8958a74a2ebe17f3f4e5346e24 P 95a0516e6b0540bcb414fedc938f21bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95a0516e6b0540bcb414fedc938f21bb, State: Initialized, Role: FOLLOWER
I20260620 19:10:04.773953 12078 raft_consensus.cc:729] T 68f1ae8958a74a2ebe17f3f4e5346e24 P 694679b53ba0487495655430a0617582 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 694679b53ba0487495655430a0617582, State: Initialized, Role: FOLLOWER
I20260620 19:10:04.774067 12082 consensus_queue.cc:260] T 68f1ae8958a74a2ebe17f3f4e5346e24 P 95a0516e6b0540bcb414fedc938f21bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1d9954fe514ae18445ac0250dae2d8" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 43623 } } peers { permanent_uuid: "95a0516e6b0540bcb414fedc938f21bb" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40669 } } peers { permanent_uuid: "694679b53ba0487495655430a0617582" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34297 } }
I20260620 19:10:04.774067 12078 consensus_queue.cc:260] T 68f1ae8958a74a2ebe17f3f4e5346e24 P 694679b53ba0487495655430a0617582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1d9954fe514ae18445ac0250dae2d8" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 43623 } } peers { permanent_uuid: "95a0516e6b0540bcb414fedc938f21bb" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40669 } } peers { permanent_uuid: "694679b53ba0487495655430a0617582" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34297 } }
I20260620 19:10:04.774111 12077 raft_consensus.cc:348] T 68f1ae8958a74a2ebe17f3f4e5346e24 P be1d9954fe514ae18445ac0250dae2d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1d9954fe514ae18445ac0250dae2d8" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 43623 } } peers { permanent_uuid: "95a0516e6b0540bcb414fedc938f21bb" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40669 } } peers { permanent_uuid: "694679b53ba0487495655430a0617582" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34297 } }
I20260620 19:10:04.774230 12077 raft_consensus.cc:374] T 68f1ae8958a74a2ebe17f3f4e5346e24 P be1d9954fe514ae18445ac0250dae2d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:04.774268 12077 raft_consensus.cc:729] T 68f1ae8958a74a2ebe17f3f4e5346e24 P be1d9954fe514ae18445ac0250dae2d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be1d9954fe514ae18445ac0250dae2d8, State: Initialized, Role: FOLLOWER
I20260620 19:10:04.774338 12082 ts_tablet_manager.cc:1428] T 68f1ae8958a74a2ebe17f3f4e5346e24 P 95a0516e6b0540bcb414fedc938f21bb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:10:04.774370 12085 tablet_replica.cc:439] T 68f1ae8958a74a2ebe17f3f4e5346e24 P 95a0516e6b0540bcb414fedc938f21bb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1d9954fe514ae18445ac0250dae2d8" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 43623 } } peers { permanent_uuid: "95a0516e6b0540bcb414fedc938f21bb" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40669 } } peers { permanent_uuid: "694679b53ba0487495655430a0617582" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34297 } } }
I20260620 19:10:04.774400 12077 consensus_queue.cc:260] T 68f1ae8958a74a2ebe17f3f4e5346e24 P be1d9954fe514ae18445ac0250dae2d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1d9954fe514ae18445ac0250dae2d8" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 43623 } } peers { permanent_uuid: "95a0516e6b0540bcb414fedc938f21bb" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40669 } } peers { permanent_uuid: "694679b53ba0487495655430a0617582" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34297 } }
I20260620 19:10:04.774500 12085 tablet_replica.cc:442] T 68f1ae8958a74a2ebe17f3f4e5346e24 P 95a0516e6b0540bcb414fedc938f21bb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:10:04.774639 12077 ts_tablet_manager.cc:1428] T 68f1ae8958a74a2ebe17f3f4e5346e24 P be1d9954fe514ae18445ac0250dae2d8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:10:04.774725 12081 tablet_replica.cc:439] T 68f1ae8958a74a2ebe17f3f4e5346e24 P be1d9954fe514ae18445ac0250dae2d8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1d9954fe514ae18445ac0250dae2d8" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 43623 } } peers { permanent_uuid: "95a0516e6b0540bcb414fedc938f21bb" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40669 } } peers { permanent_uuid: "694679b53ba0487495655430a0617582" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34297 } } }
I20260620 19:10:04.774853 12081 tablet_replica.cc:442] T 68f1ae8958a74a2ebe17f3f4e5346e24 P be1d9954fe514ae18445ac0250dae2d8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:10:04.775080 12083 tablet_replica.cc:439] T 68f1ae8958a74a2ebe17f3f4e5346e24 P 694679b53ba0487495655430a0617582: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1d9954fe514ae18445ac0250dae2d8" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 43623 } } peers { permanent_uuid: "95a0516e6b0540bcb414fedc938f21bb" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40669 } } peers { permanent_uuid: "694679b53ba0487495655430a0617582" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34297 } } }
I20260620 19:10:04.775203 12083 tablet_replica.cc:442] T 68f1ae8958a74a2ebe17f3f4e5346e24 P 694679b53ba0487495655430a0617582: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:10:04.776412 12078 ts_tablet_manager.cc:1428] T 68f1ae8958a74a2ebe17f3f4e5346e24 P 694679b53ba0487495655430a0617582: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260620 19:10:04.836486 12083 raft_consensus.cc:482] T 68f1ae8958a74a2ebe17f3f4e5346e24 P 694679b53ba0487495655430a0617582 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:10:04.836623 12083 raft_consensus.cc:504] T 68f1ae8958a74a2ebe17f3f4e5346e24 P 694679b53ba0487495655430a0617582 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1d9954fe514ae18445ac0250dae2d8" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 43623 } } peers { permanent_uuid: "95a0516e6b0540bcb414fedc938f21bb" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40669 } } peers { permanent_uuid: "694679b53ba0487495655430a0617582" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34297 } }
I20260620 19:10:04.837030 12083 leader_election.cc:288] T 68f1ae8958a74a2ebe17f3f4e5346e24 P 694679b53ba0487495655430a0617582 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers be1d9954fe514ae18445ac0250dae2d8 (127.9.155.65:43623), 95a0516e6b0540bcb414fedc938f21bb (127.9.155.66:40669)
I20260620 19:10:04.837522 11868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68f1ae8958a74a2ebe17f3f4e5346e24" candidate_uuid: "694679b53ba0487495655430a0617582" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95a0516e6b0540bcb414fedc938f21bb" is_pre_election: true
I20260620 19:10:04.837520 11719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68f1ae8958a74a2ebe17f3f4e5346e24" candidate_uuid: "694679b53ba0487495655430a0617582" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be1d9954fe514ae18445ac0250dae2d8" is_pre_election: true
I20260620 19:10:04.837666 11868 raft_consensus.cc:2454] T 68f1ae8958a74a2ebe17f3f4e5346e24 P 95a0516e6b0540bcb414fedc938f21bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 694679b53ba0487495655430a0617582 in term 0.
I20260620 19:10:04.837684 11719 raft_consensus.cc:2454] T 68f1ae8958a74a2ebe17f3f4e5346e24 P be1d9954fe514ae18445ac0250dae2d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 694679b53ba0487495655430a0617582 in term 0.
I20260620 19:10:04.838001 11939 leader_election.cc:302] T 68f1ae8958a74a2ebe17f3f4e5346e24 P 694679b53ba0487495655430a0617582 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 694679b53ba0487495655430a0617582, 95a0516e6b0540bcb414fedc938f21bb; no voters:
I20260620 19:10:04.838186 12083 raft_consensus.cc:2781] T 68f1ae8958a74a2ebe17f3f4e5346e24 P 694679b53ba0487495655430a0617582 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:10:04.838248 12083 raft_consensus.cc:482] T 68f1ae8958a74a2ebe17f3f4e5346e24 P 694679b53ba0487495655430a0617582 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:10:04.838274 12083 raft_consensus.cc:3037] T 68f1ae8958a74a2ebe17f3f4e5346e24 P 694679b53ba0487495655430a0617582 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:04.839263 12083 raft_consensus.cc:504] T 68f1ae8958a74a2ebe17f3f4e5346e24 P 694679b53ba0487495655430a0617582 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1d9954fe514ae18445ac0250dae2d8" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 43623 } } peers { permanent_uuid: "95a0516e6b0540bcb414fedc938f21bb" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40669 } } peers { permanent_uuid: "694679b53ba0487495655430a0617582" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34297 } }
I20260620 19:10:04.839646 12083 leader_election.cc:288] T 68f1ae8958a74a2ebe17f3f4e5346e24 P 694679b53ba0487495655430a0617582 [CANDIDATE]: Term 1 election: Requested vote from peers be1d9954fe514ae18445ac0250dae2d8 (127.9.155.65:43623), 95a0516e6b0540bcb414fedc938f21bb (127.9.155.66:40669)
I20260620 19:10:04.840099 11719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68f1ae8958a74a2ebe17f3f4e5346e24" candidate_uuid: "694679b53ba0487495655430a0617582" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be1d9954fe514ae18445ac0250dae2d8"
I20260620 19:10:04.840099 11868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68f1ae8958a74a2ebe17f3f4e5346e24" candidate_uuid: "694679b53ba0487495655430a0617582" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95a0516e6b0540bcb414fedc938f21bb"
I20260620 19:10:04.840272 11868 raft_consensus.cc:3037] T 68f1ae8958a74a2ebe17f3f4e5346e24 P 95a0516e6b0540bcb414fedc938f21bb [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:04.840272 11719 raft_consensus.cc:3037] T 68f1ae8958a74a2ebe17f3f4e5346e24 P be1d9954fe514ae18445ac0250dae2d8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:04.841593 11719 raft_consensus.cc:2454] T 68f1ae8958a74a2ebe17f3f4e5346e24 P be1d9954fe514ae18445ac0250dae2d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 694679b53ba0487495655430a0617582 in term 1.
I20260620 19:10:04.841593 11868 raft_consensus.cc:2454] T 68f1ae8958a74a2ebe17f3f4e5346e24 P 95a0516e6b0540bcb414fedc938f21bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 694679b53ba0487495655430a0617582 in term 1.
I20260620 19:10:04.842010 11940 leader_election.cc:302] T 68f1ae8958a74a2ebe17f3f4e5346e24 P 694679b53ba0487495655430a0617582 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 694679b53ba0487495655430a0617582, be1d9954fe514ae18445ac0250dae2d8; no voters:
I20260620 19:10:04.842199 12083 raft_consensus.cc:2781] T 68f1ae8958a74a2ebe17f3f4e5346e24 P 694679b53ba0487495655430a0617582 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:10:04.842288 12083 raft_consensus.cc:686] T 68f1ae8958a74a2ebe17f3f4e5346e24 P 694679b53ba0487495655430a0617582 [term 1 LEADER]: Becoming Leader. State: Replica: 694679b53ba0487495655430a0617582, State: Running, Role: LEADER
I20260620 19:10:04.842435 12083 consensus_queue.cc:237] T 68f1ae8958a74a2ebe17f3f4e5346e24 P 694679b53ba0487495655430a0617582 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1d9954fe514ae18445ac0250dae2d8" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 43623 } } peers { permanent_uuid: "95a0516e6b0540bcb414fedc938f21bb" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40669 } } peers { permanent_uuid: "694679b53ba0487495655430a0617582" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34297 } }
I20260620 19:10:04.843210 12083 tablet_replica.cc:439] T 68f1ae8958a74a2ebe17f3f4e5346e24 P 694679b53ba0487495655430a0617582: TxnStatusTablet state changed. Reason: New leader 694679b53ba0487495655430a0617582. Latest consensus state: current_term: 1 leader_uuid: "694679b53ba0487495655430a0617582" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1d9954fe514ae18445ac0250dae2d8" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 43623 } } peers { permanent_uuid: "95a0516e6b0540bcb414fedc938f21bb" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40669 } } peers { permanent_uuid: "694679b53ba0487495655430a0617582" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34297 } } }
I20260620 19:10:04.843317 12083 tablet_replica.cc:442] T 68f1ae8958a74a2ebe17f3f4e5346e24 P 694679b53ba0487495655430a0617582: This TxnStatusTablet replica's current role is: LEADER
I20260620 19:10:04.843796 12099 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 19:10:04.844034 12099 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 19:10:04.844494 11569 catalog_manager.cc:5495] T 68f1ae8958a74a2ebe17f3f4e5346e24 P 694679b53ba0487495655430a0617582 reported cstate change: term changed from 0 to 1, leader changed from <none> to 694679b53ba0487495655430a0617582 (127.9.155.67). New cstate: current_term: 1 leader_uuid: "694679b53ba0487495655430a0617582" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1d9954fe514ae18445ac0250dae2d8" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 43623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95a0516e6b0540bcb414fedc938f21bb" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40669 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "694679b53ba0487495655430a0617582" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34297 } health_report { overall_health: HEALTHY } } }
I20260620 19:10:04.860864 12094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:04.857147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.67:34297 (local address 127.0.0.1:44424)
0620 19:10:04.857398 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:04.857413 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:04.857531 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:04.857829 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:04.857832 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:04.857839 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 19:10:04.857940 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:04.857946 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:04.858666 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:04.858669 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:04.859276 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:04.859283 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:04.859691 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:04.859695 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:04.859789 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:04.859846 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 19:10:04.860557 (+ 711us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 19:10:04.860574 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 19:10:04.860639 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 19:10:04.860732 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142}
I20260620 19:10:04.860864 12076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:04.857233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44424 (local address 127.9.155.67:34297)
0620 19:10:04.857390 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:04.857394 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:04.857464 (+ 70us) server_negotiation.cc:407] Connection header received
0620 19:10:04.857663 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:04.857666 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:04.857720 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:04.857796 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:10:04.858038 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:04.858564 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:04.859389 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:04.859598 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:04.859681 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:04.859985 (+ 304us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:10:04.860469 (+ 484us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:10:04.860577 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:04.860649 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 19:10:04.860711 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 19:10:04.865962 11868 raft_consensus.cc:1261] T 68f1ae8958a74a2ebe17f3f4e5346e24 P 95a0516e6b0540bcb414fedc938f21bb [term 1 FOLLOWER]: Refusing update from remote peer 694679b53ba0487495655430a0617582: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:10:04.865994 11719 raft_consensus.cc:1261] T 68f1ae8958a74a2ebe17f3f4e5346e24 P be1d9954fe514ae18445ac0250dae2d8 [term 1 FOLLOWER]: Refusing update from remote peer 694679b53ba0487495655430a0617582: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:10:04.866842 12083 consensus_queue.cc:1035] T 68f1ae8958a74a2ebe17f3f4e5346e24 P 694679b53ba0487495655430a0617582 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95a0516e6b0540bcb414fedc938f21bb" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:10:04.867347 12083 consensus_queue.cc:1035] T 68f1ae8958a74a2ebe17f3f4e5346e24 P 694679b53ba0487495655430a0617582 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be1d9954fe514ae18445ac0250dae2d8" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 43623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:10:04.871233 12085 tablet_replica.cc:439] T 68f1ae8958a74a2ebe17f3f4e5346e24 P 95a0516e6b0540bcb414fedc938f21bb: TxnStatusTablet state changed. Reason: New leader 694679b53ba0487495655430a0617582. Latest consensus state: current_term: 1 leader_uuid: "694679b53ba0487495655430a0617582" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1d9954fe514ae18445ac0250dae2d8" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 43623 } } peers { permanent_uuid: "95a0516e6b0540bcb414fedc938f21bb" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40669 } } peers { permanent_uuid: "694679b53ba0487495655430a0617582" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34297 } } }
I20260620 19:10:04.871381 12085 tablet_replica.cc:442] T 68f1ae8958a74a2ebe17f3f4e5346e24 P 95a0516e6b0540bcb414fedc938f21bb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:10:04.872916 12081 tablet_replica.cc:439] T 68f1ae8958a74a2ebe17f3f4e5346e24 P be1d9954fe514ae18445ac0250dae2d8: TxnStatusTablet state changed. Reason: New leader 694679b53ba0487495655430a0617582. Latest consensus state: current_term: 1 leader_uuid: "694679b53ba0487495655430a0617582" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1d9954fe514ae18445ac0250dae2d8" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 43623 } } peers { permanent_uuid: "95a0516e6b0540bcb414fedc938f21bb" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40669 } } peers { permanent_uuid: "694679b53ba0487495655430a0617582" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34297 } } }
I20260620 19:10:04.873067 12081 tablet_replica.cc:442] T 68f1ae8958a74a2ebe17f3f4e5346e24 P be1d9954fe514ae18445ac0250dae2d8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:10:04.875198 12087 tablet_replica.cc:439] T 68f1ae8958a74a2ebe17f3f4e5346e24 P 694679b53ba0487495655430a0617582: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "694679b53ba0487495655430a0617582" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1d9954fe514ae18445ac0250dae2d8" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 43623 } } peers { permanent_uuid: "95a0516e6b0540bcb414fedc938f21bb" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40669 } } peers { permanent_uuid: "694679b53ba0487495655430a0617582" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34297 } } }
I20260620 19:10:04.875331 12087 tablet_replica.cc:442] T 68f1ae8958a74a2ebe17f3f4e5346e24 P 694679b53ba0487495655430a0617582: This TxnStatusTablet replica's current role is: LEADER
I20260620 19:10:04.875381 12085 tablet_replica.cc:439] T 68f1ae8958a74a2ebe17f3f4e5346e24 P 95a0516e6b0540bcb414fedc938f21bb: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "694679b53ba0487495655430a0617582" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1d9954fe514ae18445ac0250dae2d8" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 43623 } } peers { permanent_uuid: "95a0516e6b0540bcb414fedc938f21bb" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40669 } } peers { permanent_uuid: "694679b53ba0487495655430a0617582" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34297 } } }
I20260620 19:10:04.875501 12085 tablet_replica.cc:442] T 68f1ae8958a74a2ebe17f3f4e5346e24 P 95a0516e6b0540bcb414fedc938f21bb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:10:04.875862 12102 mvcc.cc:205] Tried to move back new op lower bound from 7299000749524156416 to 7299000749436223488. Current Snapshot: MvccSnapshot[applied={T|T < 7299000749524156416}]
I20260620 19:10:04.877014 12083 tablet_replica.cc:439] T 68f1ae8958a74a2ebe17f3f4e5346e24 P 694679b53ba0487495655430a0617582: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "694679b53ba0487495655430a0617582" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1d9954fe514ae18445ac0250dae2d8" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 43623 } } peers { permanent_uuid: "95a0516e6b0540bcb414fedc938f21bb" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40669 } } peers { permanent_uuid: "694679b53ba0487495655430a0617582" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34297 } } }
I20260620 19:10:04.877125 12083 tablet_replica.cc:442] T 68f1ae8958a74a2ebe17f3f4e5346e24 P 694679b53ba0487495655430a0617582: This TxnStatusTablet replica's current role is: LEADER
I20260620 19:10:04.878173 12103 mvcc.cc:205] Tried to move back new op lower bound from 7299000749524156416 to 7299000749436223488. Current Snapshot: MvccSnapshot[applied={T|T < 7299000749524156416}]
I20260620 19:10:04.878427 12081 tablet_replica.cc:439] T 68f1ae8958a74a2ebe17f3f4e5346e24 P be1d9954fe514ae18445ac0250dae2d8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "694679b53ba0487495655430a0617582" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1d9954fe514ae18445ac0250dae2d8" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 43623 } } peers { permanent_uuid: "95a0516e6b0540bcb414fedc938f21bb" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40669 } } peers { permanent_uuid: "694679b53ba0487495655430a0617582" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34297 } } }
I20260620 19:10:04.878539 12081 tablet_replica.cc:442] T 68f1ae8958a74a2ebe17f3f4e5346e24 P be1d9954fe514ae18445ac0250dae2d8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:10:04.879890 12105 mvcc.cc:205] Tried to move back new op lower bound from 7299000749524156416 to 7299000749436223488. Current Snapshot: MvccSnapshot[applied={T|T < 7299000749524156416}]
I20260620 19:10:04.887681 12076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:04.884577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44436 (local address 127.9.155.67:34297)
0620 19:10:04.884709 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:04.884714 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:04.884727 (+ 13us) server_negotiation.cc:407] Connection header received
0620 19:10:04.884895 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:04.884897 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:04.884941 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:04.885001 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:10:04.885248 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:04.885841 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:04.886711 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:04.886933 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:04.887016 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:04.887263 (+ 247us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:10:04.887339 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:10:04.887396 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:04.887486 (+ 90us) server_negotiation.cc:299] Negotiation successful
0620 19:10:04.887540 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 19:10:04.899185 12060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:04.895574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.67:34297 (local address 127.9.155.67:35523)
0620 19:10:04.895717 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:04.895732 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:04.895829 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:04.896049 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:04.896051 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:04.896057 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 19:10:04.896154 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:04.896159 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:04.897160 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:04.897163 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:04.897883 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:04.897894 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:04.898487 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:04.898489 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:04.898562 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:04.898613 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 19:10:04.898920 (+ 307us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 19:10:04.898925 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 19:10:04.898973 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 19:10:04.899048 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 19:10:04.899209 12076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:04.895611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.67:35523 (local address 127.9.155.67:34297)
0620 19:10:04.895720 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:04.895725 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:04.895801 (+ 76us) server_negotiation.cc:407] Connection header received
0620 19:10:04.895899 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:04.895902 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:04.895954 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:04.896030 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:10:04.896260 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:04.897039 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:04.898004 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:04.898337 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:04.898445 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:04.898752 (+ 307us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:10:04.898840 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:10:04.898907 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:04.898998 (+ 91us) server_negotiation.cc:299] Negotiation successful
0620 19:10:04.899059 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":44}
I20260620 19:10:04.905866 11719 raft_consensus.cc:1261] T 293e3d9f325a4239822a43e01f439391 P be1d9954fe514ae18445ac0250dae2d8 [term 1 FOLLOWER]: Refusing update from remote peer 694679b53ba0487495655430a0617582: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:10:04.905927 11868 raft_consensus.cc:1261] T 293e3d9f325a4239822a43e01f439391 P 95a0516e6b0540bcb414fedc938f21bb [term 1 FOLLOWER]: Refusing update from remote peer 694679b53ba0487495655430a0617582: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:10:04.906313 12083 consensus_queue.cc:1035] T 293e3d9f325a4239822a43e01f439391 P 694679b53ba0487495655430a0617582 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95a0516e6b0540bcb414fedc938f21bb" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:10:04.906486 12087 consensus_queue.cc:1035] T 293e3d9f325a4239822a43e01f439391 P 694679b53ba0487495655430a0617582 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be1d9954fe514ae18445ac0250dae2d8" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 43623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:10:04.955307 11571 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53092:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:10:04.955505 11571 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53092:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:10:04.958137 11571 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4d94d0235626484baae6753c95f8e3cf: Sending DeleteTablet for 3 replicas of tablet 293e3d9f325a4239822a43e01f439391
I20260620 19:10:04.958856 11848 tablet_service.cc:1514] Processing DeleteTablet for tablet 293e3d9f325a4239822a43e01f439391 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:10:04 UTC) from {username='kudu', principal='kudu/127.9.155.126@KRBTEST.COM'} at 127.0.0.1:47168
I20260620 19:10:04.958907 11708 tablet_service.cc:1514] Processing DeleteTablet for tablet 293e3d9f325a4239822a43e01f439391 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:10:04 UTC) from {username='kudu', principal='kudu/127.9.155.126@KRBTEST.COM'} at 127.0.0.1:40316
I20260620 19:10:04.959348 12118 tablet_replica.cc:330] T 293e3d9f325a4239822a43e01f439391 P be1d9954fe514ae18445ac0250dae2d8: stopping tablet replica
I20260620 19:10:04.959705 12119 tablet_replica.cc:330] T 293e3d9f325a4239822a43e01f439391 P 95a0516e6b0540bcb414fedc938f21bb: stopping tablet replica
I20260620 19:10:04.959707 12118 raft_consensus.cc:2229] T 293e3d9f325a4239822a43e01f439391 P be1d9954fe514ae18445ac0250dae2d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:10:04.960021 12119 raft_consensus.cc:2229] T 293e3d9f325a4239822a43e01f439391 P 95a0516e6b0540bcb414fedc938f21bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:10:04.960028 11987 tablet_service.cc:1514] Processing DeleteTablet for tablet 293e3d9f325a4239822a43e01f439391 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:10:04 UTC) from {username='kudu', principal='kudu/127.9.155.126@KRBTEST.COM'} at 127.0.0.1:44422
I20260620 19:10:04.960050 12118 raft_consensus.cc:2258] T 293e3d9f325a4239822a43e01f439391 P be1d9954fe514ae18445ac0250dae2d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:10:04.960270 12119 raft_consensus.cc:2258] T 293e3d9f325a4239822a43e01f439391 P 95a0516e6b0540bcb414fedc938f21bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:10:04.960781 12120 tablet_replica.cc:330] T 293e3d9f325a4239822a43e01f439391 P 694679b53ba0487495655430a0617582: stopping tablet replica
I20260620 19:10:04.960793 11571 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53092:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 19:10:04.961045 12120 raft_consensus.cc:2229] T 293e3d9f325a4239822a43e01f439391 P 694679b53ba0487495655430a0617582 [term 1 LEADER]: Raft consensus shutting down.
W20260620 19:10:04.961208 11571 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:10:04.961403 12120 raft_consensus.cc:2258] T 293e3d9f325a4239822a43e01f439391 P 694679b53ba0487495655430a0617582 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:10:04.962212 12118 ts_tablet_manager.cc:1905] T 293e3d9f325a4239822a43e01f439391 P be1d9954fe514ae18445ac0250dae2d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:10:04.962527 12120 ts_tablet_manager.cc:1905] T 293e3d9f325a4239822a43e01f439391 P 694679b53ba0487495655430a0617582: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:10:04.965559 12118 ts_tablet_manager.cc:1918] T 293e3d9f325a4239822a43e01f439391 P be1d9954fe514ae18445ac0250dae2d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 19:10:04.965576 12120 ts_tablet_manager.cc:1918] T 293e3d9f325a4239822a43e01f439391 P 694679b53ba0487495655430a0617582: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 19:10:04.965656 12120 log.cc:1196] T 293e3d9f325a4239822a43e01f439391 P 694679b53ba0487495655430a0617582: Deleting WAL directory at /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/minicluster-data/ts-2/wal/wals/293e3d9f325a4239822a43e01f439391
I20260620 19:10:04.965772 12118 log.cc:1196] T 293e3d9f325a4239822a43e01f439391 P be1d9954fe514ae18445ac0250dae2d8: Deleting WAL directory at /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/minicluster-data/ts-0/wal/wals/293e3d9f325a4239822a43e01f439391
I20260620 19:10:04.966066 12120 ts_tablet_manager.cc:1939] T 293e3d9f325a4239822a43e01f439391 P 694679b53ba0487495655430a0617582: Deleting consensus metadata
I20260620 19:10:04.966239 12118 ts_tablet_manager.cc:1939] T 293e3d9f325a4239822a43e01f439391 P be1d9954fe514ae18445ac0250dae2d8: Deleting consensus metadata
I20260620 19:10:04.967173 11708 tablet_service.cc:1467] Processing CreateTablet for tablet 771fcc7c35764f5e861630b28867692d (DEFAULT_TABLE table=test-table [id=d40a205d4c904df1886f967232fbbf7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:10:04.967434 11708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 771fcc7c35764f5e861630b28867692d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:04.967443 11556 catalog_manager.cc:4841] TS be1d9954fe514ae18445ac0250dae2d8 (127.9.155.65:43623): tablet 293e3d9f325a4239822a43e01f439391 (table test-table [id=e4b70625f7eb4d64b515906c87c719cb]) successfully deleted
I20260620 19:10:04.968493 11987 tablet_service.cc:1467] Processing CreateTablet for tablet 771fcc7c35764f5e861630b28867692d (DEFAULT_TABLE table=test-table [id=d40a205d4c904df1886f967232fbbf7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:10:04.968739 11987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 771fcc7c35764f5e861630b28867692d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:04.969494 12077 tablet_bootstrap.cc:492] T 771fcc7c35764f5e861630b28867692d P be1d9954fe514ae18445ac0250dae2d8: Bootstrap starting.
I20260620 19:10:04.970461 12077 tablet_bootstrap.cc:654] T 771fcc7c35764f5e861630b28867692d P be1d9954fe514ae18445ac0250dae2d8: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:04.970994 12078 tablet_bootstrap.cc:492] T 771fcc7c35764f5e861630b28867692d P 694679b53ba0487495655430a0617582: Bootstrap starting.
I20260620 19:10:04.971155 11557 catalog_manager.cc:4841] TS 694679b53ba0487495655430a0617582 (127.9.155.67:34297): tablet 293e3d9f325a4239822a43e01f439391 (table test-table [id=e4b70625f7eb4d64b515906c87c719cb]) successfully deleted
I20260620 19:10:04.971668 11848 tablet_service.cc:1467] Processing CreateTablet for tablet 771fcc7c35764f5e861630b28867692d (DEFAULT_TABLE table=test-table [id=d40a205d4c904df1886f967232fbbf7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:10:04.971913 12078 tablet_bootstrap.cc:654] T 771fcc7c35764f5e861630b28867692d P 694679b53ba0487495655430a0617582: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:04.971940 11848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 771fcc7c35764f5e861630b28867692d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:04.972854 12119 ts_tablet_manager.cc:1905] T 293e3d9f325a4239822a43e01f439391 P 95a0516e6b0540bcb414fedc938f21bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:10:04.973857 12077 tablet_bootstrap.cc:492] T 771fcc7c35764f5e861630b28867692d P be1d9954fe514ae18445ac0250dae2d8: No bootstrap required, opened a new log
I20260620 19:10:04.973948 12077 ts_tablet_manager.cc:1397] T 771fcc7c35764f5e861630b28867692d P be1d9954fe514ae18445ac0250dae2d8: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 19:10:04.974465 12082 tablet_bootstrap.cc:492] T 771fcc7c35764f5e861630b28867692d P 95a0516e6b0540bcb414fedc938f21bb: Bootstrap starting.
I20260620 19:10:04.974432 12077 raft_consensus.cc:348] T 771fcc7c35764f5e861630b28867692d P be1d9954fe514ae18445ac0250dae2d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1d9954fe514ae18445ac0250dae2d8" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 43623 } } peers { permanent_uuid: "95a0516e6b0540bcb414fedc938f21bb" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40669 } } peers { permanent_uuid: "694679b53ba0487495655430a0617582" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34297 } }
I20260620 19:10:04.974553 12077 raft_consensus.cc:374] T 771fcc7c35764f5e861630b28867692d P be1d9954fe514ae18445ac0250dae2d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:04.974582 12077 raft_consensus.cc:729] T 771fcc7c35764f5e861630b28867692d P be1d9954fe514ae18445ac0250dae2d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be1d9954fe514ae18445ac0250dae2d8, State: Initialized, Role: FOLLOWER
I20260620 19:10:04.974700 12077 consensus_queue.cc:260] T 771fcc7c35764f5e861630b28867692d P be1d9954fe514ae18445ac0250dae2d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1d9954fe514ae18445ac0250dae2d8" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 43623 } } peers { permanent_uuid: "95a0516e6b0540bcb414fedc938f21bb" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40669 } } peers { permanent_uuid: "694679b53ba0487495655430a0617582" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34297 } }
I20260620 19:10:04.974979 12077 ts_tablet_manager.cc:1428] T 771fcc7c35764f5e861630b28867692d P be1d9954fe514ae18445ac0250dae2d8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:10:04.975411 12082 tablet_bootstrap.cc:654] T 771fcc7c35764f5e861630b28867692d P 95a0516e6b0540bcb414fedc938f21bb: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:04.977177 12078 tablet_bootstrap.cc:492] T 771fcc7c35764f5e861630b28867692d P 694679b53ba0487495655430a0617582: No bootstrap required, opened a new log
I20260620 19:10:04.977283 12078 ts_tablet_manager.cc:1397] T 771fcc7c35764f5e861630b28867692d P 694679b53ba0487495655430a0617582: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.002s
I20260620 19:10:04.977761 12078 raft_consensus.cc:348] T 771fcc7c35764f5e861630b28867692d P 694679b53ba0487495655430a0617582 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1d9954fe514ae18445ac0250dae2d8" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 43623 } } peers { permanent_uuid: "95a0516e6b0540bcb414fedc938f21bb" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40669 } } peers { permanent_uuid: "694679b53ba0487495655430a0617582" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34297 } }
I20260620 19:10:04.977896 12078 raft_consensus.cc:374] T 771fcc7c35764f5e861630b28867692d P 694679b53ba0487495655430a0617582 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:04.977931 12078 raft_consensus.cc:729] T 771fcc7c35764f5e861630b28867692d P 694679b53ba0487495655430a0617582 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 694679b53ba0487495655430a0617582, State: Initialized, Role: FOLLOWER
I20260620 19:10:04.978091 12078 consensus_queue.cc:260] T 771fcc7c35764f5e861630b28867692d P 694679b53ba0487495655430a0617582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1d9954fe514ae18445ac0250dae2d8" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 43623 } } peers { permanent_uuid: "95a0516e6b0540bcb414fedc938f21bb" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40669 } } peers { permanent_uuid: "694679b53ba0487495655430a0617582" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34297 } }
I20260620 19:10:04.978369 12078 ts_tablet_manager.cc:1428] T 771fcc7c35764f5e861630b28867692d P 694679b53ba0487495655430a0617582: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:10:04.978794 12119 ts_tablet_manager.cc:1918] T 293e3d9f325a4239822a43e01f439391 P 95a0516e6b0540bcb414fedc938f21bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 19:10:04.978899 12119 log.cc:1196] T 293e3d9f325a4239822a43e01f439391 P 95a0516e6b0540bcb414fedc938f21bb: Deleting WAL directory at /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/minicluster-data/ts-1/wal/wals/293e3d9f325a4239822a43e01f439391
I20260620 19:10:04.979197 12119 ts_tablet_manager.cc:1939] T 293e3d9f325a4239822a43e01f439391 P 95a0516e6b0540bcb414fedc938f21bb: Deleting consensus metadata
I20260620 19:10:04.979465 12082 tablet_bootstrap.cc:492] T 771fcc7c35764f5e861630b28867692d P 95a0516e6b0540bcb414fedc938f21bb: No bootstrap required, opened a new log
I20260620 19:10:04.979549 12082 ts_tablet_manager.cc:1397] T 771fcc7c35764f5e861630b28867692d P 95a0516e6b0540bcb414fedc938f21bb: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260620 19:10:04.979948 12082 raft_consensus.cc:348] T 771fcc7c35764f5e861630b28867692d P 95a0516e6b0540bcb414fedc938f21bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1d9954fe514ae18445ac0250dae2d8" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 43623 } } peers { permanent_uuid: "95a0516e6b0540bcb414fedc938f21bb" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40669 } } peers { permanent_uuid: "694679b53ba0487495655430a0617582" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34297 } }
I20260620 19:10:04.980059 12082 raft_consensus.cc:374] T 771fcc7c35764f5e861630b28867692d P 95a0516e6b0540bcb414fedc938f21bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:04.980090 12082 raft_consensus.cc:729] T 771fcc7c35764f5e861630b28867692d P 95a0516e6b0540bcb414fedc938f21bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95a0516e6b0540bcb414fedc938f21bb, State: Initialized, Role: FOLLOWER
I20260620 19:10:04.980199 12082 consensus_queue.cc:260] T 771fcc7c35764f5e861630b28867692d P 95a0516e6b0540bcb414fedc938f21bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1d9954fe514ae18445ac0250dae2d8" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 43623 } } peers { permanent_uuid: "95a0516e6b0540bcb414fedc938f21bb" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40669 } } peers { permanent_uuid: "694679b53ba0487495655430a0617582" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34297 } }
I20260620 19:10:04.980485 12082 ts_tablet_manager.cc:1428] T 771fcc7c35764f5e861630b28867692d P 95a0516e6b0540bcb414fedc938f21bb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:10:04.980468 11555 catalog_manager.cc:4841] TS 95a0516e6b0540bcb414fedc938f21bb (127.9.155.66:40669): tablet 293e3d9f325a4239822a43e01f439391 (table test-table [id=e4b70625f7eb4d64b515906c87c719cb]) successfully deleted
I20260620 19:10:05.316092 12081 raft_consensus.cc:482] T 771fcc7c35764f5e861630b28867692d P be1d9954fe514ae18445ac0250dae2d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:10:05.316290 12081 raft_consensus.cc:504] T 771fcc7c35764f5e861630b28867692d P be1d9954fe514ae18445ac0250dae2d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1d9954fe514ae18445ac0250dae2d8" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 43623 } } peers { permanent_uuid: "95a0516e6b0540bcb414fedc938f21bb" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40669 } } peers { permanent_uuid: "694679b53ba0487495655430a0617582" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34297 } }
I20260620 19:10:05.317579 12081 leader_election.cc:288] T 771fcc7c35764f5e861630b28867692d P be1d9954fe514ae18445ac0250dae2d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 95a0516e6b0540bcb414fedc938f21bb (127.9.155.66:40669), 694679b53ba0487495655430a0617582 (127.9.155.67:34297)
I20260620 19:10:05.317759 12115 raft_consensus.cc:482] T 771fcc7c35764f5e861630b28867692d P 694679b53ba0487495655430a0617582 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:10:05.317914 12115 raft_consensus.cc:504] T 771fcc7c35764f5e861630b28867692d P 694679b53ba0487495655430a0617582 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1d9954fe514ae18445ac0250dae2d8" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 43623 } } peers { permanent_uuid: "95a0516e6b0540bcb414fedc938f21bb" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40669 } } peers { permanent_uuid: "694679b53ba0487495655430a0617582" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34297 } }
I20260620 19:10:05.318280 12115 leader_election.cc:288] T 771fcc7c35764f5e861630b28867692d P 694679b53ba0487495655430a0617582 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers be1d9954fe514ae18445ac0250dae2d8 (127.9.155.65:43623), 95a0516e6b0540bcb414fedc938f21bb (127.9.155.66:40669)
I20260620 19:10:05.318732 11719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "771fcc7c35764f5e861630b28867692d" candidate_uuid: "694679b53ba0487495655430a0617582" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be1d9954fe514ae18445ac0250dae2d8" is_pre_election: true
I20260620 19:10:05.318753 11868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "771fcc7c35764f5e861630b28867692d" candidate_uuid: "694679b53ba0487495655430a0617582" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95a0516e6b0540bcb414fedc938f21bb" is_pre_election: true
I20260620 19:10:05.318857 11719 raft_consensus.cc:2454] T 771fcc7c35764f5e861630b28867692d P be1d9954fe514ae18445ac0250dae2d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 694679b53ba0487495655430a0617582 in term 0.
I20260620 19:10:05.318902 11868 raft_consensus.cc:2454] T 771fcc7c35764f5e861630b28867692d P 95a0516e6b0540bcb414fedc938f21bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 694679b53ba0487495655430a0617582 in term 0.
I20260620 19:10:05.319228 11939 leader_election.cc:302] T 771fcc7c35764f5e861630b28867692d P 694679b53ba0487495655430a0617582 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 694679b53ba0487495655430a0617582, 95a0516e6b0540bcb414fedc938f21bb; no voters:
I20260620 19:10:05.319514 12115 raft_consensus.cc:2781] T 771fcc7c35764f5e861630b28867692d P 694679b53ba0487495655430a0617582 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:10:05.319589 12115 raft_consensus.cc:482] T 771fcc7c35764f5e861630b28867692d P 694679b53ba0487495655430a0617582 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:10:05.319631 12115 raft_consensus.cc:3037] T 771fcc7c35764f5e861630b28867692d P 694679b53ba0487495655430a0617582 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:05.320832 12115 raft_consensus.cc:504] T 771fcc7c35764f5e861630b28867692d P 694679b53ba0487495655430a0617582 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1d9954fe514ae18445ac0250dae2d8" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 43623 } } peers { permanent_uuid: "95a0516e6b0540bcb414fedc938f21bb" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40669 } } peers { permanent_uuid: "694679b53ba0487495655430a0617582" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34297 } }
I20260620 19:10:05.321188 12115 leader_election.cc:288] T 771fcc7c35764f5e861630b28867692d P 694679b53ba0487495655430a0617582 [CANDIDATE]: Term 1 election: Requested vote from peers be1d9954fe514ae18445ac0250dae2d8 (127.9.155.65:43623), 95a0516e6b0540bcb414fedc938f21bb (127.9.155.66:40669)
I20260620 19:10:05.321599 11719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "771fcc7c35764f5e861630b28867692d" candidate_uuid: "694679b53ba0487495655430a0617582" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be1d9954fe514ae18445ac0250dae2d8"
I20260620 19:10:05.321607 11868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "771fcc7c35764f5e861630b28867692d" candidate_uuid: "694679b53ba0487495655430a0617582" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95a0516e6b0540bcb414fedc938f21bb"
I20260620 19:10:05.321720 11719 raft_consensus.cc:3037] T 771fcc7c35764f5e861630b28867692d P be1d9954fe514ae18445ac0250dae2d8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:05.321725 11868 raft_consensus.cc:3037] T 771fcc7c35764f5e861630b28867692d P 95a0516e6b0540bcb414fedc938f21bb [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:05.322244 12124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:05.317863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.66:40669 (local address 127.9.155.65:40685)
0620 19:10:05.318375 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:05.318389 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:05.318507 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:05.318843 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:05.318846 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:05.318863 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:10:05.318966 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:05.318973 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:05.319919 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:05.319922 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:05.320923 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:05.320932 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:05.321845 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:05.321848 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:05.321932 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:05.321955 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 19:10:05.322021 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 19:10:05.322092 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"mutex_wait_us":23,"thread_start_us":80,"threads_started":1}
I20260620 19:10:05.322649 12126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:05.318017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.65:40685 (local address 127.9.155.66:40669)
0620 19:10:05.318308 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:05.318312 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:05.318488 (+ 176us) server_negotiation.cc:407] Connection header received
0620 19:10:05.318591 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:05.318594 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:05.318641 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:05.318718 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:10:05.319083 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:05.319797 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:05.321060 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:05.321608 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:05.321715 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:05.321788 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:05.322443 (+ 655us) server_negotiation.cc:299] Negotiation successful
0620 19:10:05.322527 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":79,"threads_started":1}
I20260620 19:10:05.322791 12076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:05.318749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.65:57857 (local address 127.9.155.67:34297)
0620 19:10:05.318863 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:05.318868 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:05.318882 (+ 14us) server_negotiation.cc:407] Connection header received
0620 19:10:05.318922 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:05.318924 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:05.318966 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:05.319037 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:10:05.319456 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:05.320197 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:05.321584 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:05.322120 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:05.322206 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:05.322250 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:05.322599 (+ 349us) server_negotiation.cc:299] Negotiation successful
0620 19:10:05.322678 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 19:10:05.322862 11719 raft_consensus.cc:2454] T 771fcc7c35764f5e861630b28867692d P be1d9954fe514ae18445ac0250dae2d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 694679b53ba0487495655430a0617582 in term 1.
I20260620 19:10:05.322873 11868 raft_consensus.cc:2454] T 771fcc7c35764f5e861630b28867692d P 95a0516e6b0540bcb414fedc938f21bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 694679b53ba0487495655430a0617582 in term 1.
I20260620 19:10:05.323118 12125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:05.317863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.67:34297 (local address 127.9.155.65:57857)
0620 19:10:05.318199 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:05.318215 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:05.318325 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:05.319163 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:05.319166 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:05.319182 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:10:05.319289 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:05.319295 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:05.320336 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:05.320340 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:05.321458 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:05.321468 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:05.322367 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:05.322371 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:05.322464 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:05.322487 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 19:10:05.322910 (+ 423us) client_negotiation.cc:240] Negotiation successful
0620 19:10:05.322986 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":72,"threads_started":1}
I20260620 19:10:05.323184 11940 leader_election.cc:302] T 771fcc7c35764f5e861630b28867692d P 694679b53ba0487495655430a0617582 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 694679b53ba0487495655430a0617582, 95a0516e6b0540bcb414fedc938f21bb, be1d9954fe514ae18445ac0250dae2d8; no voters:
I20260620 19:10:05.323370 12115 raft_consensus.cc:2781] T 771fcc7c35764f5e861630b28867692d P 694679b53ba0487495655430a0617582 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:10:05.323459 12115 raft_consensus.cc:686] T 771fcc7c35764f5e861630b28867692d P 694679b53ba0487495655430a0617582 [term 1 LEADER]: Becoming Leader. State: Replica: 694679b53ba0487495655430a0617582, State: Running, Role: LEADER
I20260620 19:10:05.323562 12008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "771fcc7c35764f5e861630b28867692d" candidate_uuid: "be1d9954fe514ae18445ac0250dae2d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "694679b53ba0487495655430a0617582" is_pre_election: true
I20260620 19:10:05.323586 12115 consensus_queue.cc:237] T 771fcc7c35764f5e861630b28867692d P 694679b53ba0487495655430a0617582 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1d9954fe514ae18445ac0250dae2d8" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 43623 } } peers { permanent_uuid: "95a0516e6b0540bcb414fedc938f21bb" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40669 } } peers { permanent_uuid: "694679b53ba0487495655430a0617582" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34297 } }
I20260620 19:10:05.323794 11868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "771fcc7c35764f5e861630b28867692d" candidate_uuid: "be1d9954fe514ae18445ac0250dae2d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95a0516e6b0540bcb414fedc938f21bb" is_pre_election: true
I20260620 19:10:05.323985 11868 raft_consensus.cc:2379] T 771fcc7c35764f5e861630b28867692d P 95a0516e6b0540bcb414fedc938f21bb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate be1d9954fe514ae18445ac0250dae2d8 in current term 1: Already voted for candidate 694679b53ba0487495655430a0617582 in this term.
I20260620 19:10:05.324402 11659 leader_election.cc:302] T 771fcc7c35764f5e861630b28867692d P be1d9954fe514ae18445ac0250dae2d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be1d9954fe514ae18445ac0250dae2d8; no voters: 694679b53ba0487495655430a0617582, 95a0516e6b0540bcb414fedc938f21bb
I20260620 19:10:05.324646 12081 raft_consensus.cc:2726] T 771fcc7c35764f5e861630b28867692d P be1d9954fe514ae18445ac0250dae2d8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 19:10:05.325282 11569 catalog_manager.cc:5495] T 771fcc7c35764f5e861630b28867692d P 694679b53ba0487495655430a0617582 reported cstate change: term changed from 0 to 1, leader changed from <none> to 694679b53ba0487495655430a0617582 (127.9.155.67). New cstate: current_term: 1 leader_uuid: "694679b53ba0487495655430a0617582" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1d9954fe514ae18445ac0250dae2d8" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 43623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95a0516e6b0540bcb414fedc938f21bb" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40669 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "694679b53ba0487495655430a0617582" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34297 } health_report { overall_health: HEALTHY } } }
I20260620 19:10:05.361337 11868 raft_consensus.cc:1261] T 771fcc7c35764f5e861630b28867692d P 95a0516e6b0540bcb414fedc938f21bb [term 1 FOLLOWER]: Refusing update from remote peer 694679b53ba0487495655430a0617582: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:10:05.361708 11719 raft_consensus.cc:1261] T 771fcc7c35764f5e861630b28867692d P be1d9954fe514ae18445ac0250dae2d8 [term 1 FOLLOWER]: Refusing update from remote peer 694679b53ba0487495655430a0617582: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:10:05.361721 12115 consensus_queue.cc:1035] T 771fcc7c35764f5e861630b28867692d P 694679b53ba0487495655430a0617582 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95a0516e6b0540bcb414fedc938f21bb" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:10:05.362092 12115 consensus_queue.cc:1035] T 771fcc7c35764f5e861630b28867692d P 694679b53ba0487495655430a0617582 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be1d9954fe514ae18445ac0250dae2d8" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 43623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 19:10:05.376367 12102 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 19:10:06.926797 11569 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53092:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:10:06.926981 11569 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53092:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:10:06.929736 11569 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4d94d0235626484baae6753c95f8e3cf: Sending DeleteTablet for 3 replicas of tablet 771fcc7c35764f5e861630b28867692d
I20260620 19:10:06.930378 11987 tablet_service.cc:1514] Processing DeleteTablet for tablet 771fcc7c35764f5e861630b28867692d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:10:06 UTC) from {username='kudu', principal='kudu/127.9.155.126@KRBTEST.COM'} at 127.0.0.1:44422
I20260620 19:10:06.930378 11848 tablet_service.cc:1514] Processing DeleteTablet for tablet 771fcc7c35764f5e861630b28867692d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:10:06 UTC) from {username='kudu', principal='kudu/127.9.155.126@KRBTEST.COM'} at 127.0.0.1:47168
I20260620 19:10:06.930389 11708 tablet_service.cc:1514] Processing DeleteTablet for tablet 771fcc7c35764f5e861630b28867692d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:10:06 UTC) from {username='kudu', principal='kudu/127.9.155.126@KRBTEST.COM'} at 127.0.0.1:40316
I20260620 19:10:06.930757 12151 tablet_replica.cc:330] T 771fcc7c35764f5e861630b28867692d P be1d9954fe514ae18445ac0250dae2d8: stopping tablet replica
I20260620 19:10:06.930898 12151 raft_consensus.cc:2229] T 771fcc7c35764f5e861630b28867692d P be1d9954fe514ae18445ac0250dae2d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:10:06.931025 12151 raft_consensus.cc:2258] T 771fcc7c35764f5e861630b28867692d P be1d9954fe514ae18445ac0250dae2d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:10:06.931656 12151 ts_tablet_manager.cc:1905] T 771fcc7c35764f5e861630b28867692d P be1d9954fe514ae18445ac0250dae2d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:10:06.934091 12149 tablet_replica.cc:330] T 771fcc7c35764f5e861630b28867692d P 694679b53ba0487495655430a0617582: stopping tablet replica
I20260620 19:10:06.934235 12149 raft_consensus.cc:2229] T 771fcc7c35764f5e861630b28867692d P 694679b53ba0487495655430a0617582 [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:10:06.934376 12151 ts_tablet_manager.cc:1918] T 771fcc7c35764f5e861630b28867692d P be1d9954fe514ae18445ac0250dae2d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 19:10:06.934446 12151 log.cc:1196] T 771fcc7c35764f5e861630b28867692d P be1d9954fe514ae18445ac0250dae2d8: Deleting WAL directory at /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/minicluster-data/ts-0/wal/wals/771fcc7c35764f5e861630b28867692d
I20260620 19:10:06.934484 12149 raft_consensus.cc:2258] T 771fcc7c35764f5e861630b28867692d P 694679b53ba0487495655430a0617582 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:10:06.934689 12151 ts_tablet_manager.cc:1939] T 771fcc7c35764f5e861630b28867692d P be1d9954fe514ae18445ac0250dae2d8: Deleting consensus metadata
I20260620 19:10:06.934736 12150 tablet_replica.cc:330] T 771fcc7c35764f5e861630b28867692d P 95a0516e6b0540bcb414fedc938f21bb: stopping tablet replica
I20260620 19:10:06.934861 12150 raft_consensus.cc:2229] T 771fcc7c35764f5e861630b28867692d P 95a0516e6b0540bcb414fedc938f21bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:10:06.934981 12150 raft_consensus.cc:2258] T 771fcc7c35764f5e861630b28867692d P 95a0516e6b0540bcb414fedc938f21bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:10:06.935079 12149 ts_tablet_manager.cc:1905] T 771fcc7c35764f5e861630b28867692d P 694679b53ba0487495655430a0617582: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:10:06.935412 11556 catalog_manager.cc:4841] TS be1d9954fe514ae18445ac0250dae2d8 (127.9.155.65:43623): tablet 771fcc7c35764f5e861630b28867692d (table test-table [id=d40a205d4c904df1886f967232fbbf7c]) successfully deleted
I20260620 19:10:06.935571 12150 ts_tablet_manager.cc:1905] T 771fcc7c35764f5e861630b28867692d P 95a0516e6b0540bcb414fedc938f21bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:10:06.937654 12149 ts_tablet_manager.cc:1918] T 771fcc7c35764f5e861630b28867692d P 694679b53ba0487495655430a0617582: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 19:10:06.937716 12149 log.cc:1196] T 771fcc7c35764f5e861630b28867692d P 694679b53ba0487495655430a0617582: Deleting WAL directory at /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/minicluster-data/ts-2/wal/wals/771fcc7c35764f5e861630b28867692d
I20260620 19:10:06.937966 12149 ts_tablet_manager.cc:1939] T 771fcc7c35764f5e861630b28867692d P 694679b53ba0487495655430a0617582: Deleting consensus metadata
I20260620 19:10:06.938930 11557 catalog_manager.cc:4841] TS 694679b53ba0487495655430a0617582 (127.9.155.67:34297): tablet 771fcc7c35764f5e861630b28867692d (table test-table [id=d40a205d4c904df1886f967232fbbf7c]) successfully deleted
I20260620 19:10:06.939375 12150 ts_tablet_manager.cc:1918] T 771fcc7c35764f5e861630b28867692d P 95a0516e6b0540bcb414fedc938f21bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 19:10:06.939430 12150 log.cc:1196] T 771fcc7c35764f5e861630b28867692d P 95a0516e6b0540bcb414fedc938f21bb: Deleting WAL directory at /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/minicluster-data/ts-1/wal/wals/771fcc7c35764f5e861630b28867692d
I20260620 19:10:06.939584 12150 ts_tablet_manager.cc:1939] T 771fcc7c35764f5e861630b28867692d P 95a0516e6b0540bcb414fedc938f21bb: Deleting consensus metadata
I20260620 19:10:06.940245 11555 catalog_manager.cc:4841] TS 95a0516e6b0540bcb414fedc938f21bb (127.9.155.66:40669): tablet 771fcc7c35764f5e861630b28867692d (table test-table [id=d40a205d4c904df1886f967232fbbf7c]) successfully deleted
Jun 20 19:10:06 dist-test-slave-n3fx krb5kdc[11526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982606, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 19:10:06 dist-test-slave-n3fx krb5kdc[11526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982606, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:10:06.959460 12159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:06.950357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53110 (local address 127.9.155.126:37761)
0620 19:10:06.950632 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:06.950638 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:06.950653 (+ 15us) server_negotiation.cc:407] Connection header received
0620 19:10:06.950722 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:06.950724 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:06.950764 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:06.950840 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:06.951532 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:06.952136 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:06.953026 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:06.953311 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:06.953400 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:06.955855 (+ 2455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:06.955859 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:06.955861 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:06.955888 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:06.957748 (+ 1860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:06.958236 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:06.958239 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:06.958241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:06.958377 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:06.958662 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:06.958664 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:06.958665 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:06.958849 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:06.958950 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:06.959183 (+ 233us) server_negotiation.cc:299] Negotiation successful
0620 19:10:06.959285 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":99,"threads_started":1}
I20260620 19:10:06.962201 11569 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53110:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:10:06.962652 11569 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:10:06.967554 11848 tablet_service.cc:1467] Processing CreateTablet for tablet b43c5f7b059b46698d6f6c557fc0173b (DEFAULT_TABLE table=test-table [id=48cecbed14aa4e7ba7bd138a6fd1d371]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:10:06.967553 11708 tablet_service.cc:1467] Processing CreateTablet for tablet b43c5f7b059b46698d6f6c557fc0173b (DEFAULT_TABLE table=test-table [id=48cecbed14aa4e7ba7bd138a6fd1d371]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:10:06.967689 11987 tablet_service.cc:1467] Processing CreateTablet for tablet b43c5f7b059b46698d6f6c557fc0173b (DEFAULT_TABLE table=test-table [id=48cecbed14aa4e7ba7bd138a6fd1d371]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:10:06.967877 11708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b43c5f7b059b46698d6f6c557fc0173b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:06.967880 11848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b43c5f7b059b46698d6f6c557fc0173b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:06.967913 11987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b43c5f7b059b46698d6f6c557fc0173b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:06.969900 12160 tablet_bootstrap.cc:492] T b43c5f7b059b46698d6f6c557fc0173b P be1d9954fe514ae18445ac0250dae2d8: Bootstrap starting.
I20260620 19:10:06.970561 12162 tablet_bootstrap.cc:492] T b43c5f7b059b46698d6f6c557fc0173b P 95a0516e6b0540bcb414fedc938f21bb: Bootstrap starting.
I20260620 19:10:06.970575 12161 tablet_bootstrap.cc:492] T b43c5f7b059b46698d6f6c557fc0173b P 694679b53ba0487495655430a0617582: Bootstrap starting.
I20260620 19:10:06.970803 12160 tablet_bootstrap.cc:654] T b43c5f7b059b46698d6f6c557fc0173b P be1d9954fe514ae18445ac0250dae2d8: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:06.971468 12161 tablet_bootstrap.cc:654] T b43c5f7b059b46698d6f6c557fc0173b P 694679b53ba0487495655430a0617582: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:06.971877 12160 tablet_bootstrap.cc:492] T b43c5f7b059b46698d6f6c557fc0173b P be1d9954fe514ae18445ac0250dae2d8: No bootstrap required, opened a new log
I20260620 19:10:06.971915 12162 tablet_bootstrap.cc:654] T b43c5f7b059b46698d6f6c557fc0173b P 95a0516e6b0540bcb414fedc938f21bb: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:06.971959 12160 ts_tablet_manager.cc:1397] T b43c5f7b059b46698d6f6c557fc0173b P be1d9954fe514ae18445ac0250dae2d8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:10:06.972420 12160 raft_consensus.cc:348] T b43c5f7b059b46698d6f6c557fc0173b P be1d9954fe514ae18445ac0250dae2d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1d9954fe514ae18445ac0250dae2d8" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 43623 } } peers { permanent_uuid: "95a0516e6b0540bcb414fedc938f21bb" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40669 } } peers { permanent_uuid: "694679b53ba0487495655430a0617582" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34297 } }
I20260620 19:10:06.972539 12160 raft_consensus.cc:374] T b43c5f7b059b46698d6f6c557fc0173b P be1d9954fe514ae18445ac0250dae2d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:06.972573 12160 raft_consensus.cc:729] T b43c5f7b059b46698d6f6c557fc0173b P be1d9954fe514ae18445ac0250dae2d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be1d9954fe514ae18445ac0250dae2d8, State: Initialized, Role: FOLLOWER
I20260620 19:10:06.972610 12161 tablet_bootstrap.cc:492] T b43c5f7b059b46698d6f6c557fc0173b P 694679b53ba0487495655430a0617582: No bootstrap required, opened a new log
I20260620 19:10:06.972679 12161 ts_tablet_manager.cc:1397] T b43c5f7b059b46698d6f6c557fc0173b P 694679b53ba0487495655430a0617582: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:10:06.972725 12160 consensus_queue.cc:260] T b43c5f7b059b46698d6f6c557fc0173b P be1d9954fe514ae18445ac0250dae2d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1d9954fe514ae18445ac0250dae2d8" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 43623 } } peers { permanent_uuid: "95a0516e6b0540bcb414fedc938f21bb" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40669 } } peers { permanent_uuid: "694679b53ba0487495655430a0617582" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34297 } }
I20260620 19:10:06.973186 12160 ts_tablet_manager.cc:1428] T b43c5f7b059b46698d6f6c557fc0173b P be1d9954fe514ae18445ac0250dae2d8: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260620 19:10:06.973130 12161 raft_consensus.cc:348] T b43c5f7b059b46698d6f6c557fc0173b P 694679b53ba0487495655430a0617582 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1d9954fe514ae18445ac0250dae2d8" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 43623 } } peers { permanent_uuid: "95a0516e6b0540bcb414fedc938f21bb" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40669 } } peers { permanent_uuid: "694679b53ba0487495655430a0617582" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34297 } }
I20260620 19:10:06.973230 12161 raft_consensus.cc:374] T b43c5f7b059b46698d6f6c557fc0173b P 694679b53ba0487495655430a0617582 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:06.973259 12161 raft_consensus.cc:729] T b43c5f7b059b46698d6f6c557fc0173b P 694679b53ba0487495655430a0617582 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 694679b53ba0487495655430a0617582, State: Initialized, Role: FOLLOWER
I20260620 19:10:06.973423 12162 tablet_bootstrap.cc:492] T b43c5f7b059b46698d6f6c557fc0173b P 95a0516e6b0540bcb414fedc938f21bb: No bootstrap required, opened a new log
I20260620 19:10:06.973387 12161 consensus_queue.cc:260] T b43c5f7b059b46698d6f6c557fc0173b P 694679b53ba0487495655430a0617582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1d9954fe514ae18445ac0250dae2d8" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 43623 } } peers { permanent_uuid: "95a0516e6b0540bcb414fedc938f21bb" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40669 } } peers { permanent_uuid: "694679b53ba0487495655430a0617582" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34297 } }
I20260620 19:10:06.973516 12162 ts_tablet_manager.cc:1397] T b43c5f7b059b46698d6f6c557fc0173b P 95a0516e6b0540bcb414fedc938f21bb: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260620 19:10:06.973640 12161 ts_tablet_manager.cc:1428] T b43c5f7b059b46698d6f6c557fc0173b P 694679b53ba0487495655430a0617582: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:10:06.974372 12162 raft_consensus.cc:348] T b43c5f7b059b46698d6f6c557fc0173b P 95a0516e6b0540bcb414fedc938f21bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1d9954fe514ae18445ac0250dae2d8" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 43623 } } peers { permanent_uuid: "95a0516e6b0540bcb414fedc938f21bb" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40669 } } peers { permanent_uuid: "694679b53ba0487495655430a0617582" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34297 } }
I20260620 19:10:06.974498 12162 raft_consensus.cc:374] T b43c5f7b059b46698d6f6c557fc0173b P 95a0516e6b0540bcb414fedc938f21bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:06.974535 12162 raft_consensus.cc:729] T b43c5f7b059b46698d6f6c557fc0173b P 95a0516e6b0540bcb414fedc938f21bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95a0516e6b0540bcb414fedc938f21bb, State: Initialized, Role: FOLLOWER
I20260620 19:10:06.974648 12162 consensus_queue.cc:260] T b43c5f7b059b46698d6f6c557fc0173b P 95a0516e6b0540bcb414fedc938f21bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1d9954fe514ae18445ac0250dae2d8" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 43623 } } peers { permanent_uuid: "95a0516e6b0540bcb414fedc938f21bb" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40669 } } peers { permanent_uuid: "694679b53ba0487495655430a0617582" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34297 } }
I20260620 19:10:06.974987 12162 ts_tablet_manager.cc:1428] T b43c5f7b059b46698d6f6c557fc0173b P 95a0516e6b0540bcb414fedc938f21bb: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 19:10:07.011029 12167 raft_consensus.cc:482] T b43c5f7b059b46698d6f6c557fc0173b P 95a0516e6b0540bcb414fedc938f21bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:10:07.011163 12167 raft_consensus.cc:504] T b43c5f7b059b46698d6f6c557fc0173b P 95a0516e6b0540bcb414fedc938f21bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1d9954fe514ae18445ac0250dae2d8" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 43623 } } peers { permanent_uuid: "95a0516e6b0540bcb414fedc938f21bb" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40669 } } peers { permanent_uuid: "694679b53ba0487495655430a0617582" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34297 } }
I20260620 19:10:07.012182 12167 leader_election.cc:288] T b43c5f7b059b46698d6f6c557fc0173b P 95a0516e6b0540bcb414fedc938f21bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers be1d9954fe514ae18445ac0250dae2d8 (127.9.155.65:43623), 694679b53ba0487495655430a0617582 (127.9.155.67:34297)
I20260620 19:10:07.016076 12171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:07.012614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.66:52821 (local address 127.9.155.67:34297)
0620 19:10:07.012898 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:07.012906 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:07.012921 (+ 15us) server_negotiation.cc:407] Connection header received
0620 19:10:07.013011 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:07.013014 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:07.013055 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:07.013117 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:10:07.013431 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:07.013960 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:07.015120 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:07.015541 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:07.015650 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:07.015706 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:07.015869 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 19:10:07.015940 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":87,"threads_started":1}
I20260620 19:10:07.016084 12170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:07.012487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.66:60447 (local address 127.9.155.65:43623)
0620 19:10:07.012781 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:07.012790 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:07.012843 (+ 53us) server_negotiation.cc:407] Connection header received
0620 19:10:07.013033 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:07.013035 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:07.013073 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:07.013150 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:10:07.013398 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:07.013981 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:07.015130 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:07.015562 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:07.015672 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:07.015734 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:07.015882 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 19:10:07.015953 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":100,"threads_started":1}
I20260620 19:10:07.016094 12169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:07.012500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.67:34297 (local address 127.9.155.66:52821)
0620 19:10:07.012795 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:07.012808 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:07.012923 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:07.013147 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:07.013149 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:07.013164 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:10:07.013322 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:07.013328 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:07.014058 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:07.014061 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:07.014994 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:07.015003 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:07.015651 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:07.015654 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:07.015749 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:07.015772 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 19:10:07.015835 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 19:10:07.015916 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"mutex_wait_us":26,"spinlock_wait_cycles":10368,"thread_start_us":97,"threads_started":1}
I20260620 19:10:07.016093 12168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:07.012400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.65:43623 (local address 127.9.155.66:60447)
0620 19:10:07.012767 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:07.012783 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:07.012924 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:07.013167 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:07.013170 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:07.013194 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:10:07.013303 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:07.013308 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:07.014114 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:07.014117 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:07.014999 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:07.015008 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:07.015675 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:07.015679 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:07.015776 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:07.015798 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 19:10:07.015860 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 19:10:07.015931 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"mutex_wait_us":32,"spinlock_wait_cycles":39552,"thread_start_us":125,"threads_started":1}
I20260620 19:10:07.016526 12008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b43c5f7b059b46698d6f6c557fc0173b" candidate_uuid: "95a0516e6b0540bcb414fedc938f21bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "694679b53ba0487495655430a0617582" is_pre_election: true
I20260620 19:10:07.016714 12008 raft_consensus.cc:2454] T b43c5f7b059b46698d6f6c557fc0173b P 694679b53ba0487495655430a0617582 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95a0516e6b0540bcb414fedc938f21bb in term 0.
I20260620 19:10:07.016903 11728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b43c5f7b059b46698d6f6c557fc0173b" candidate_uuid: "95a0516e6b0540bcb414fedc938f21bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be1d9954fe514ae18445ac0250dae2d8" is_pre_election: true
I20260620 19:10:07.017045 11728 raft_consensus.cc:2454] T b43c5f7b059b46698d6f6c557fc0173b P be1d9954fe514ae18445ac0250dae2d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95a0516e6b0540bcb414fedc938f21bb in term 0.
I20260620 19:10:07.017122 11801 leader_election.cc:302] T b43c5f7b059b46698d6f6c557fc0173b P 95a0516e6b0540bcb414fedc938f21bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 694679b53ba0487495655430a0617582, 95a0516e6b0540bcb414fedc938f21bb; no voters:
I20260620 19:10:07.017382 12167 raft_consensus.cc:2781] T b43c5f7b059b46698d6f6c557fc0173b P 95a0516e6b0540bcb414fedc938f21bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:10:07.017436 12167 raft_consensus.cc:482] T b43c5f7b059b46698d6f6c557fc0173b P 95a0516e6b0540bcb414fedc938f21bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:10:07.017465 12167 raft_consensus.cc:3037] T b43c5f7b059b46698d6f6c557fc0173b P 95a0516e6b0540bcb414fedc938f21bb [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:07.018455 12167 raft_consensus.cc:504] T b43c5f7b059b46698d6f6c557fc0173b P 95a0516e6b0540bcb414fedc938f21bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1d9954fe514ae18445ac0250dae2d8" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 43623 } } peers { permanent_uuid: "95a0516e6b0540bcb414fedc938f21bb" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40669 } } peers { permanent_uuid: "694679b53ba0487495655430a0617582" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34297 } }
I20260620 19:10:07.018816 12167 leader_election.cc:288] T b43c5f7b059b46698d6f6c557fc0173b P 95a0516e6b0540bcb414fedc938f21bb [CANDIDATE]: Term 1 election: Requested vote from peers be1d9954fe514ae18445ac0250dae2d8 (127.9.155.65:43623), 694679b53ba0487495655430a0617582 (127.9.155.67:34297)
I20260620 19:10:07.019207 12008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b43c5f7b059b46698d6f6c557fc0173b" candidate_uuid: "95a0516e6b0540bcb414fedc938f21bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "694679b53ba0487495655430a0617582"
I20260620 19:10:07.019354 12008 raft_consensus.cc:3037] T b43c5f7b059b46698d6f6c557fc0173b P 694679b53ba0487495655430a0617582 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:07.019433 11728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b43c5f7b059b46698d6f6c557fc0173b" candidate_uuid: "95a0516e6b0540bcb414fedc938f21bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be1d9954fe514ae18445ac0250dae2d8"
I20260620 19:10:07.019573 11728 raft_consensus.cc:3037] T b43c5f7b059b46698d6f6c557fc0173b P be1d9954fe514ae18445ac0250dae2d8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:07.020519 12008 raft_consensus.cc:2454] T b43c5f7b059b46698d6f6c557fc0173b P 694679b53ba0487495655430a0617582 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95a0516e6b0540bcb414fedc938f21bb in term 1.
I20260620 19:10:07.020749 11728 raft_consensus.cc:2454] T b43c5f7b059b46698d6f6c557fc0173b P be1d9954fe514ae18445ac0250dae2d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95a0516e6b0540bcb414fedc938f21bb in term 1.
I20260620 19:10:07.020850 11801 leader_election.cc:302] T b43c5f7b059b46698d6f6c557fc0173b P 95a0516e6b0540bcb414fedc938f21bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 694679b53ba0487495655430a0617582, 95a0516e6b0540bcb414fedc938f21bb; no voters:
I20260620 19:10:07.021030 12167 raft_consensus.cc:2781] T b43c5f7b059b46698d6f6c557fc0173b P 95a0516e6b0540bcb414fedc938f21bb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:10:07.021291 12167 raft_consensus.cc:686] T b43c5f7b059b46698d6f6c557fc0173b P 95a0516e6b0540bcb414fedc938f21bb [term 1 LEADER]: Becoming Leader. State: Replica: 95a0516e6b0540bcb414fedc938f21bb, State: Running, Role: LEADER
I20260620 19:10:07.021543 12167 consensus_queue.cc:237] T b43c5f7b059b46698d6f6c557fc0173b P 95a0516e6b0540bcb414fedc938f21bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1d9954fe514ae18445ac0250dae2d8" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 43623 } } peers { permanent_uuid: "95a0516e6b0540bcb414fedc938f21bb" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40669 } } peers { permanent_uuid: "694679b53ba0487495655430a0617582" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34297 } }
I20260620 19:10:07.024072 11569 catalog_manager.cc:5495] T b43c5f7b059b46698d6f6c557fc0173b P 95a0516e6b0540bcb414fedc938f21bb reported cstate change: term changed from 0 to 1, leader changed from <none> to 95a0516e6b0540bcb414fedc938f21bb (127.9.155.66). New cstate: current_term: 1 leader_uuid: "95a0516e6b0540bcb414fedc938f21bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1d9954fe514ae18445ac0250dae2d8" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 43623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95a0516e6b0540bcb414fedc938f21bb" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40669 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "694679b53ba0487495655430a0617582" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34297 } health_report { overall_health: UNKNOWN } } }
I20260620 19:10:07.047789 12175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:07.044159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47176 (local address 127.9.155.66:40669)
0620 19:10:07.044477 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:07.044485 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:07.044498 (+ 13us) server_negotiation.cc:407] Connection header received
0620 19:10:07.044574 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:07.044577 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:07.044628 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:07.044736 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:10:07.044980 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:07.045536 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:07.046377 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:07.046606 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:07.046690 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:07.046950 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:10:07.047398 (+ 448us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:10:07.047500 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:07.047570 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 19:10:07.047626 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":141,"threads_started":1}
I20260620 19:10:07.058495 12178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:07.054952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.67:34297 (local address 127.9.155.66:57875)
0620 19:10:07.055243 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:07.055256 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:07.055357 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:07.055615 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:07.055619 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:07.055627 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 19:10:07.055745 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:07.055750 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:07.056649 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:07.056651 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:07.057310 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:07.057319 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:07.057745 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:07.057747 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:07.057818 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:07.057891 (+ 73us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 19:10:07.058216 (+ 325us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 19:10:07.058222 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 19:10:07.058272 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 19:10:07.058355 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":86,"threads_started":1}
I20260620 19:10:07.058499 12171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:07.055063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.66:57875 (local address 127.9.155.67:34297)
0620 19:10:07.055177 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:07.055181 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:07.055327 (+ 146us) server_negotiation.cc:407] Connection header received
0620 19:10:07.055436 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:07.055439 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:07.055495 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:07.055566 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:10:07.055877 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:07.056528 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:07.057429 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:07.057654 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:07.057736 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:07.058048 (+ 312us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:10:07.058129 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:10:07.058193 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:07.058333 (+ 140us) server_negotiation.cc:299] Negotiation successful
0620 19:10:07.058393 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 19:10:07.065287 11728 raft_consensus.cc:1261] T b43c5f7b059b46698d6f6c557fc0173b P be1d9954fe514ae18445ac0250dae2d8 [term 1 FOLLOWER]: Refusing update from remote peer 95a0516e6b0540bcb414fedc938f21bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:10:07.065376 12008 raft_consensus.cc:1261] T b43c5f7b059b46698d6f6c557fc0173b P 694679b53ba0487495655430a0617582 [term 1 FOLLOWER]: Refusing update from remote peer 95a0516e6b0540bcb414fedc938f21bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:10:07.065902 12172 consensus_queue.cc:1035] T b43c5f7b059b46698d6f6c557fc0173b P 95a0516e6b0540bcb414fedc938f21bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "be1d9954fe514ae18445ac0250dae2d8" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 43623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:10:07.066082 12167 consensus_queue.cc:1035] T b43c5f7b059b46698d6f6c557fc0173b P 95a0516e6b0540bcb414fedc938f21bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "694679b53ba0487495655430a0617582" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:10:07.083382 12175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:07.080309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.67:45555 (local address 127.9.155.66:40669)
0620 19:10:07.080416 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:07.080421 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:07.080571 (+ 150us) server_negotiation.cc:407] Connection header received
0620 19:10:07.080658 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:07.080662 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:07.080707 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:07.080762 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:10:07.080983 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:07.081574 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:07.082440 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:07.082647 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:07.082730 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:07.082954 (+ 224us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:10:07.083023 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:10:07.083076 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:07.083194 (+ 118us) server_negotiation.cc:299] Negotiation successful
0620 19:10:07.083249 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 19:10:07.083416 12183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:07.080153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.66:40669 (local address 127.9.155.67:45555)
0620 19:10:07.080492 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:07.080504 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:07.080585 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:07.080772 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:07.080774 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:07.080779 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 19:10:07.080864 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:07.080869 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:07.081694 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:07.081696 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:07.082344 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:07.082353 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:07.082731 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:07.082733 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:07.082801 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:07.082847 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 19:10:07.083099 (+ 252us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 19:10:07.083107 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 19:10:07.083172 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 19:10:07.083252 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":149,"threads_started":1}
I20260620 19:10:07.103992 11569 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53110:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:10:07.104197 11569 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53110:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:10:07.106676 11569 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4d94d0235626484baae6753c95f8e3cf: Sending DeleteTablet for 3 replicas of tablet b43c5f7b059b46698d6f6c557fc0173b
I20260620 19:10:07.107368 11848 tablet_service.cc:1514] Processing DeleteTablet for tablet b43c5f7b059b46698d6f6c557fc0173b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:10:07 UTC) from {username='kudu', principal='kudu/127.9.155.126@KRBTEST.COM'} at 127.0.0.1:47168
I20260620 19:10:07.107398 11987 tablet_service.cc:1514] Processing DeleteTablet for tablet b43c5f7b059b46698d6f6c557fc0173b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:10:07 UTC) from {username='kudu', principal='kudu/127.9.155.126@KRBTEST.COM'} at 127.0.0.1:44422
I20260620 19:10:07.107429 11708 tablet_service.cc:1514] Processing DeleteTablet for tablet b43c5f7b059b46698d6f6c557fc0173b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:10:07 UTC) from {username='kudu', principal='kudu/127.9.155.126@KRBTEST.COM'} at 127.0.0.1:40316
I20260620 19:10:07.107558 12150 tablet_replica.cc:330] T b43c5f7b059b46698d6f6c557fc0173b P 95a0516e6b0540bcb414fedc938f21bb: stopping tablet replica
I20260620 19:10:07.107649 12151 tablet_replica.cc:330] T b43c5f7b059b46698d6f6c557fc0173b P be1d9954fe514ae18445ac0250dae2d8: stopping tablet replica
I20260620 19:10:07.107664 12149 tablet_replica.cc:330] T b43c5f7b059b46698d6f6c557fc0173b P 694679b53ba0487495655430a0617582: stopping tablet replica
I20260620 19:10:07.107671 12150 raft_consensus.cc:2229] T b43c5f7b059b46698d6f6c557fc0173b P 95a0516e6b0540bcb414fedc938f21bb [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:10:07.107753 12151 raft_consensus.cc:2229] T b43c5f7b059b46698d6f6c557fc0173b P be1d9954fe514ae18445ac0250dae2d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:10:07.107795 12149 raft_consensus.cc:2229] T b43c5f7b059b46698d6f6c557fc0173b P 694679b53ba0487495655430a0617582 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:10:07.107869 12151 raft_consensus.cc:2258] T b43c5f7b059b46698d6f6c557fc0173b P be1d9954fe514ae18445ac0250dae2d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:10:07.107931 12149 raft_consensus.cc:2258] T b43c5f7b059b46698d6f6c557fc0173b P 694679b53ba0487495655430a0617582 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:10:07.107990 12150 raft_consensus.cc:2258] T b43c5f7b059b46698d6f6c557fc0173b P 95a0516e6b0540bcb414fedc938f21bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:10:07.108664 12150 ts_tablet_manager.cc:1905] T b43c5f7b059b46698d6f6c557fc0173b P 95a0516e6b0540bcb414fedc938f21bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:10:07.108803 12149 ts_tablet_manager.cc:1905] T b43c5f7b059b46698d6f6c557fc0173b P 694679b53ba0487495655430a0617582: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:10:07.109436 11569 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53110:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:10:07.109891 11569 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:10:07.111174 12149 ts_tablet_manager.cc:1918] T b43c5f7b059b46698d6f6c557fc0173b P 694679b53ba0487495655430a0617582: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 19:10:07.111174 12150 ts_tablet_manager.cc:1918] T b43c5f7b059b46698d6f6c557fc0173b P 95a0516e6b0540bcb414fedc938f21bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 19:10:07.111246 12149 log.cc:1196] T b43c5f7b059b46698d6f6c557fc0173b P 694679b53ba0487495655430a0617582: Deleting WAL directory at /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/minicluster-data/ts-2/wal/wals/b43c5f7b059b46698d6f6c557fc0173b
I20260620 19:10:07.111246 12150 log.cc:1196] T b43c5f7b059b46698d6f6c557fc0173b P 95a0516e6b0540bcb414fedc938f21bb: Deleting WAL directory at /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/minicluster-data/ts-1/wal/wals/b43c5f7b059b46698d6f6c557fc0173b
I20260620 19:10:07.111475 12149 ts_tablet_manager.cc:1939] T b43c5f7b059b46698d6f6c557fc0173b P 694679b53ba0487495655430a0617582: Deleting consensus metadata
I20260620 19:10:07.111475 12150 ts_tablet_manager.cc:1939] T b43c5f7b059b46698d6f6c557fc0173b P 95a0516e6b0540bcb414fedc938f21bb: Deleting consensus metadata
I20260620 19:10:07.112244 11557 catalog_manager.cc:4841] TS 694679b53ba0487495655430a0617582 (127.9.155.67:34297): tablet b43c5f7b059b46698d6f6c557fc0173b (table test-table [id=48cecbed14aa4e7ba7bd138a6fd1d371]) successfully deleted
I20260620 19:10:07.112494 12151 ts_tablet_manager.cc:1905] T b43c5f7b059b46698d6f6c557fc0173b P be1d9954fe514ae18445ac0250dae2d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:10:07.112898 11555 catalog_manager.cc:4841] TS 95a0516e6b0540bcb414fedc938f21bb (127.9.155.66:40669): tablet b43c5f7b059b46698d6f6c557fc0173b (table test-table [id=48cecbed14aa4e7ba7bd138a6fd1d371]) successfully deleted
I20260620 19:10:07.114913 12151 ts_tablet_manager.cc:1918] T b43c5f7b059b46698d6f6c557fc0173b P be1d9954fe514ae18445ac0250dae2d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 19:10:07.114980 12151 log.cc:1196] T b43c5f7b059b46698d6f6c557fc0173b P be1d9954fe514ae18445ac0250dae2d8: Deleting WAL directory at /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/minicluster-data/ts-0/wal/wals/b43c5f7b059b46698d6f6c557fc0173b
I20260620 19:10:07.115213 12151 ts_tablet_manager.cc:1939] T b43c5f7b059b46698d6f6c557fc0173b P be1d9954fe514ae18445ac0250dae2d8: Deleting consensus metadata
I20260620 19:10:07.115890 11556 catalog_manager.cc:4841] TS be1d9954fe514ae18445ac0250dae2d8 (127.9.155.65:43623): tablet b43c5f7b059b46698d6f6c557fc0173b (table test-table [id=48cecbed14aa4e7ba7bd138a6fd1d371]) successfully deleted
I20260620 19:10:07.118664 11708 tablet_service.cc:1467] Processing CreateTablet for tablet cc483552d457459499f9094afe4081d9 (DEFAULT_TABLE table=test-table [id=ed307f0e979d4e9a94381c48ffb45689]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:10:07.118705 11987 tablet_service.cc:1467] Processing CreateTablet for tablet cc483552d457459499f9094afe4081d9 (DEFAULT_TABLE table=test-table [id=ed307f0e979d4e9a94381c48ffb45689]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:10:07.118705 11848 tablet_service.cc:1467] Processing CreateTablet for tablet cc483552d457459499f9094afe4081d9 (DEFAULT_TABLE table=test-table [id=ed307f0e979d4e9a94381c48ffb45689]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:10:07.118945 11708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc483552d457459499f9094afe4081d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:07.118969 11987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc483552d457459499f9094afe4081d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:07.118969 11848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc483552d457459499f9094afe4081d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:07.121423 12162 tablet_bootstrap.cc:492] T cc483552d457459499f9094afe4081d9 P 95a0516e6b0540bcb414fedc938f21bb: Bootstrap starting.
I20260620 19:10:07.121706 12161 tablet_bootstrap.cc:492] T cc483552d457459499f9094afe4081d9 P 694679b53ba0487495655430a0617582: Bootstrap starting.
I20260620 19:10:07.121831 12160 tablet_bootstrap.cc:492] T cc483552d457459499f9094afe4081d9 P be1d9954fe514ae18445ac0250dae2d8: Bootstrap starting.
I20260620 19:10:07.122418 12162 tablet_bootstrap.cc:654] T cc483552d457459499f9094afe4081d9 P 95a0516e6b0540bcb414fedc938f21bb: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:07.122450 12161 tablet_bootstrap.cc:654] T cc483552d457459499f9094afe4081d9 P 694679b53ba0487495655430a0617582: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:07.122774 12160 tablet_bootstrap.cc:654] T cc483552d457459499f9094afe4081d9 P be1d9954fe514ae18445ac0250dae2d8: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:07.123930 12162 tablet_bootstrap.cc:492] T cc483552d457459499f9094afe4081d9 P 95a0516e6b0540bcb414fedc938f21bb: No bootstrap required, opened a new log
I20260620 19:10:07.124014 12162 ts_tablet_manager.cc:1397] T cc483552d457459499f9094afe4081d9 P 95a0516e6b0540bcb414fedc938f21bb: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260620 19:10:07.124047 12160 tablet_bootstrap.cc:492] T cc483552d457459499f9094afe4081d9 P be1d9954fe514ae18445ac0250dae2d8: No bootstrap required, opened a new log
I20260620 19:10:07.124130 12160 ts_tablet_manager.cc:1397] T cc483552d457459499f9094afe4081d9 P be1d9954fe514ae18445ac0250dae2d8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:10:07.124496 12162 raft_consensus.cc:348] T cc483552d457459499f9094afe4081d9 P 95a0516e6b0540bcb414fedc938f21bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1d9954fe514ae18445ac0250dae2d8" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 43623 } } peers { permanent_uuid: "95a0516e6b0540bcb414fedc938f21bb" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40669 } } peers { permanent_uuid: "694679b53ba0487495655430a0617582" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34297 } }
I20260620 19:10:07.124625 12162 raft_consensus.cc:374] T cc483552d457459499f9094afe4081d9 P 95a0516e6b0540bcb414fedc938f21bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:07.124586 12160 raft_consensus.cc:348] T cc483552d457459499f9094afe4081d9 P be1d9954fe514ae18445ac0250dae2d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1d9954fe514ae18445ac0250dae2d8" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 43623 } } peers { permanent_uuid: "95a0516e6b0540bcb414fedc938f21bb" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40669 } } peers { permanent_uuid: "694679b53ba0487495655430a0617582" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34297 } }
I20260620 19:10:07.124661 12162 raft_consensus.cc:729] T cc483552d457459499f9094afe4081d9 P 95a0516e6b0540bcb414fedc938f21bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95a0516e6b0540bcb414fedc938f21bb, State: Initialized, Role: FOLLOWER
I20260620 19:10:07.124691 12160 raft_consensus.cc:374] T cc483552d457459499f9094afe4081d9 P be1d9954fe514ae18445ac0250dae2d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:07.124722 12160 raft_consensus.cc:729] T cc483552d457459499f9094afe4081d9 P be1d9954fe514ae18445ac0250dae2d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be1d9954fe514ae18445ac0250dae2d8, State: Initialized, Role: FOLLOWER
I20260620 19:10:07.124776 12162 consensus_queue.cc:260] T cc483552d457459499f9094afe4081d9 P 95a0516e6b0540bcb414fedc938f21bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1d9954fe514ae18445ac0250dae2d8" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 43623 } } peers { permanent_uuid: "95a0516e6b0540bcb414fedc938f21bb" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40669 } } peers { permanent_uuid: "694679b53ba0487495655430a0617582" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34297 } }
I20260620 19:10:07.124848 12160 consensus_queue.cc:260] T cc483552d457459499f9094afe4081d9 P be1d9954fe514ae18445ac0250dae2d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1d9954fe514ae18445ac0250dae2d8" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 43623 } } peers { permanent_uuid: "95a0516e6b0540bcb414fedc938f21bb" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40669 } } peers { permanent_uuid: "694679b53ba0487495655430a0617582" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34297 } }
I20260620 19:10:07.124897 12161 tablet_bootstrap.cc:492] T cc483552d457459499f9094afe4081d9 P 694679b53ba0487495655430a0617582: No bootstrap required, opened a new log
I20260620 19:10:07.124966 12161 ts_tablet_manager.cc:1397] T cc483552d457459499f9094afe4081d9 P 694679b53ba0487495655430a0617582: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 19:10:07.125041 12162 ts_tablet_manager.cc:1428] T cc483552d457459499f9094afe4081d9 P 95a0516e6b0540bcb414fedc938f21bb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:10:07.125087 12160 ts_tablet_manager.cc:1428] T cc483552d457459499f9094afe4081d9 P be1d9954fe514ae18445ac0250dae2d8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:10:07.125337 12161 raft_consensus.cc:348] T cc483552d457459499f9094afe4081d9 P 694679b53ba0487495655430a0617582 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1d9954fe514ae18445ac0250dae2d8" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 43623 } } peers { permanent_uuid: "95a0516e6b0540bcb414fedc938f21bb" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40669 } } peers { permanent_uuid: "694679b53ba0487495655430a0617582" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34297 } }
I20260620 19:10:07.125437 12161 raft_consensus.cc:374] T cc483552d457459499f9094afe4081d9 P 694679b53ba0487495655430a0617582 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:07.125465 12161 raft_consensus.cc:729] T cc483552d457459499f9094afe4081d9 P 694679b53ba0487495655430a0617582 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 694679b53ba0487495655430a0617582, State: Initialized, Role: FOLLOWER
I20260620 19:10:07.125576 12161 consensus_queue.cc:260] T cc483552d457459499f9094afe4081d9 P 694679b53ba0487495655430a0617582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1d9954fe514ae18445ac0250dae2d8" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 43623 } } peers { permanent_uuid: "95a0516e6b0540bcb414fedc938f21bb" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40669 } } peers { permanent_uuid: "694679b53ba0487495655430a0617582" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34297 } }
I20260620 19:10:07.125846 12161 ts_tablet_manager.cc:1428] T cc483552d457459499f9094afe4081d9 P 694679b53ba0487495655430a0617582: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:10:07.169847 12172 raft_consensus.cc:482] T cc483552d457459499f9094afe4081d9 P 95a0516e6b0540bcb414fedc938f21bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:10:07.170008 12172 raft_consensus.cc:504] T cc483552d457459499f9094afe4081d9 P 95a0516e6b0540bcb414fedc938f21bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1d9954fe514ae18445ac0250dae2d8" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 43623 } } peers { permanent_uuid: "95a0516e6b0540bcb414fedc938f21bb" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40669 } } peers { permanent_uuid: "694679b53ba0487495655430a0617582" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34297 } }
I20260620 19:10:07.170472 12172 leader_election.cc:288] T cc483552d457459499f9094afe4081d9 P 95a0516e6b0540bcb414fedc938f21bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers be1d9954fe514ae18445ac0250dae2d8 (127.9.155.65:43623), 694679b53ba0487495655430a0617582 (127.9.155.67:34297)
I20260620 19:10:07.170917 11728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc483552d457459499f9094afe4081d9" candidate_uuid: "95a0516e6b0540bcb414fedc938f21bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be1d9954fe514ae18445ac0250dae2d8" is_pre_election: true
I20260620 19:10:07.170964 12008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc483552d457459499f9094afe4081d9" candidate_uuid: "95a0516e6b0540bcb414fedc938f21bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "694679b53ba0487495655430a0617582" is_pre_election: true
I20260620 19:10:07.171067 11728 raft_consensus.cc:2454] T cc483552d457459499f9094afe4081d9 P be1d9954fe514ae18445ac0250dae2d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95a0516e6b0540bcb414fedc938f21bb in term 0.
I20260620 19:10:07.171104 12008 raft_consensus.cc:2454] T cc483552d457459499f9094afe4081d9 P 694679b53ba0487495655430a0617582 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95a0516e6b0540bcb414fedc938f21bb in term 0.
I20260620 19:10:07.171442 11801 leader_election.cc:302] T cc483552d457459499f9094afe4081d9 P 95a0516e6b0540bcb414fedc938f21bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 694679b53ba0487495655430a0617582, 95a0516e6b0540bcb414fedc938f21bb; no voters:
I20260620 19:10:07.171669 12172 raft_consensus.cc:2781] T cc483552d457459499f9094afe4081d9 P 95a0516e6b0540bcb414fedc938f21bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:10:07.171739 12172 raft_consensus.cc:482] T cc483552d457459499f9094afe4081d9 P 95a0516e6b0540bcb414fedc938f21bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:10:07.171774 12172 raft_consensus.cc:3037] T cc483552d457459499f9094afe4081d9 P 95a0516e6b0540bcb414fedc938f21bb [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:07.173053 12172 raft_consensus.cc:504] T cc483552d457459499f9094afe4081d9 P 95a0516e6b0540bcb414fedc938f21bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1d9954fe514ae18445ac0250dae2d8" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 43623 } } peers { permanent_uuid: "95a0516e6b0540bcb414fedc938f21bb" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40669 } } peers { permanent_uuid: "694679b53ba0487495655430a0617582" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34297 } }
I20260620 19:10:07.173440 12172 leader_election.cc:288] T cc483552d457459499f9094afe4081d9 P 95a0516e6b0540bcb414fedc938f21bb [CANDIDATE]: Term 1 election: Requested vote from peers be1d9954fe514ae18445ac0250dae2d8 (127.9.155.65:43623), 694679b53ba0487495655430a0617582 (127.9.155.67:34297)
I20260620 19:10:07.173859 11728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc483552d457459499f9094afe4081d9" candidate_uuid: "95a0516e6b0540bcb414fedc938f21bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be1d9954fe514ae18445ac0250dae2d8"
I20260620 19:10:07.173890 12008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc483552d457459499f9094afe4081d9" candidate_uuid: "95a0516e6b0540bcb414fedc938f21bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "694679b53ba0487495655430a0617582"
I20260620 19:10:07.173983 11728 raft_consensus.cc:3037] T cc483552d457459499f9094afe4081d9 P be1d9954fe514ae18445ac0250dae2d8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:07.173997 12008 raft_consensus.cc:3037] T cc483552d457459499f9094afe4081d9 P 694679b53ba0487495655430a0617582 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:07.175153 11728 raft_consensus.cc:2454] T cc483552d457459499f9094afe4081d9 P be1d9954fe514ae18445ac0250dae2d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95a0516e6b0540bcb414fedc938f21bb in term 1.
I20260620 19:10:07.175160 12008 raft_consensus.cc:2454] T cc483552d457459499f9094afe4081d9 P 694679b53ba0487495655430a0617582 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95a0516e6b0540bcb414fedc938f21bb in term 1.
I20260620 19:10:07.175642 11800 leader_election.cc:302] T cc483552d457459499f9094afe4081d9 P 95a0516e6b0540bcb414fedc938f21bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95a0516e6b0540bcb414fedc938f21bb, be1d9954fe514ae18445ac0250dae2d8; no voters:
I20260620 19:10:07.175856 12172 raft_consensus.cc:2781] T cc483552d457459499f9094afe4081d9 P 95a0516e6b0540bcb414fedc938f21bb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:10:07.175931 12172 raft_consensus.cc:686] T cc483552d457459499f9094afe4081d9 P 95a0516e6b0540bcb414fedc938f21bb [term 1 LEADER]: Becoming Leader. State: Replica: 95a0516e6b0540bcb414fedc938f21bb, State: Running, Role: LEADER
I20260620 19:10:07.176054 12172 consensus_queue.cc:237] T cc483552d457459499f9094afe4081d9 P 95a0516e6b0540bcb414fedc938f21bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1d9954fe514ae18445ac0250dae2d8" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 43623 } } peers { permanent_uuid: "95a0516e6b0540bcb414fedc938f21bb" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40669 } } peers { permanent_uuid: "694679b53ba0487495655430a0617582" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34297 } }
I20260620 19:10:07.177772 11570 catalog_manager.cc:5495] T cc483552d457459499f9094afe4081d9 P 95a0516e6b0540bcb414fedc938f21bb reported cstate change: term changed from 0 to 1, leader changed from <none> to 95a0516e6b0540bcb414fedc938f21bb (127.9.155.66). New cstate: current_term: 1 leader_uuid: "95a0516e6b0540bcb414fedc938f21bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1d9954fe514ae18445ac0250dae2d8" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 43623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95a0516e6b0540bcb414fedc938f21bb" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40669 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "694679b53ba0487495655430a0617582" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34297 } health_report { overall_health: UNKNOWN } } }
I20260620 19:10:07.204905 11719 raft_consensus.cc:1261] T cc483552d457459499f9094afe4081d9 P be1d9954fe514ae18445ac0250dae2d8 [term 1 FOLLOWER]: Refusing update from remote peer 95a0516e6b0540bcb414fedc938f21bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:10:07.205269 12008 raft_consensus.cc:1261] T cc483552d457459499f9094afe4081d9 P 694679b53ba0487495655430a0617582 [term 1 FOLLOWER]: Refusing update from remote peer 95a0516e6b0540bcb414fedc938f21bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:10:07.205709 12184 consensus_queue.cc:1035] T cc483552d457459499f9094afe4081d9 P 95a0516e6b0540bcb414fedc938f21bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "be1d9954fe514ae18445ac0250dae2d8" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 43623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:10:07.205888 12172 consensus_queue.cc:1035] T cc483552d457459499f9094afe4081d9 P 95a0516e6b0540bcb414fedc938f21bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "694679b53ba0487495655430a0617582" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 19:10:07.220484 12133 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 19:10:08.774274 11571 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53110:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:10:08.774458 11571 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53110:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:10:08.777067 11571 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4d94d0235626484baae6753c95f8e3cf: Sending DeleteTablet for 3 replicas of tablet cc483552d457459499f9094afe4081d9
I20260620 19:10:08.777731 11708 tablet_service.cc:1514] Processing DeleteTablet for tablet cc483552d457459499f9094afe4081d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:10:08 UTC) from {username='kudu', principal='kudu/127.9.155.126@KRBTEST.COM'} at 127.0.0.1:40316
I20260620 19:10:08.778074 11848 tablet_service.cc:1514] Processing DeleteTablet for tablet cc483552d457459499f9094afe4081d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:10:08 UTC) from {username='kudu', principal='kudu/127.9.155.126@KRBTEST.COM'} at 127.0.0.1:47168
I20260620 19:10:08.778115 12214 tablet_replica.cc:330] T cc483552d457459499f9094afe4081d9 P be1d9954fe514ae18445ac0250dae2d8: stopping tablet replica
I20260620 19:10:08.778255 12214 raft_consensus.cc:2229] T cc483552d457459499f9094afe4081d9 P be1d9954fe514ae18445ac0250dae2d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:10:08.778266 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 11643
I20260620 19:10:08.778366 11987 tablet_service.cc:1514] Processing DeleteTablet for tablet cc483552d457459499f9094afe4081d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:10:08 UTC) from {username='kudu', principal='kudu/127.9.155.126@KRBTEST.COM'} at 127.0.0.1:44422
I20260620 19:10:08.779403 12216 tablet_replica.cc:330] T cc483552d457459499f9094afe4081d9 P 694679b53ba0487495655430a0617582: stopping tablet replica
I20260620 19:10:08.779510 12216 raft_consensus.cc:2229] T cc483552d457459499f9094afe4081d9 P 694679b53ba0487495655430a0617582 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:10:08.779629 12216 raft_consensus.cc:2258] T cc483552d457459499f9094afe4081d9 P 694679b53ba0487495655430a0617582 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:10:08.780313 12216 ts_tablet_manager.cc:1905] T cc483552d457459499f9094afe4081d9 P 694679b53ba0487495655430a0617582: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:10:08.782465 12216 ts_tablet_manager.cc:1918] T cc483552d457459499f9094afe4081d9 P 694679b53ba0487495655430a0617582: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 19:10:08.782534 12216 log.cc:1196] T cc483552d457459499f9094afe4081d9 P 694679b53ba0487495655430a0617582: Deleting WAL directory at /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982595231230-9837-0/minicluster-data/ts-2/wal/wals/cc483552d457459499f9094afe4081d9
I20260620 19:10:08.782728 12216 ts_tablet_manager.cc:1939] T cc483552d457459499f9094afe4081d9 P 694679b53ba0487495655430a0617582: Deleting consensus metadata
I20260620 19:10:08.782783 12215 tablet_replica.cc:330] T cc483552d457459499f9094afe4081d9 P 95a0516e6b0540bcb414fedc938f21bb: stopping tablet replica
I20260620 19:10:08.782954 12215 raft_consensus.cc:2229] T cc483552d457459499f9094afe4081d9 P 95a0516e6b0540bcb414fedc938f21bb [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:10:08.783165 12215 raft_consensus.cc:2258] T cc483552d457459499f9094afe4081d9 P 95a0516e6b0540bcb414fedc938f21bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:10:08.783468 11557 catalog_manager.cc:4841] TS 694679b53ba0487495655430a0617582 (127.9.155.67:34297): tablet cc483552d457459499f9094afe4081d9 (table test-table [id=ed307f0e979d4e9a94381c48ffb45689]) successfully deleted
I20260620 19:10:08.783732 12215 ts_tablet_manager.cc:1905] T cc483552d457459499f9094afe4081d9 P 95a0516e6b0540bcb414fedc938f21bb: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 19:10:08.784345 11556 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.155.65:43623 (error 108)
I20260620 19:10:08.785387 12218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:08.785023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.65:43623 (local address 127.0.0.1:34546)
0620 19:10:08.785235 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:08.785295 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.65:43623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":84,"threads_started":1}
I20260620 19:10:08.785537 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 11783
W20260620 19:10:08.785609 11556 catalog_manager.cc:4584] TS be1d9954fe514ae18445ac0250dae2d8 (127.9.155.65:43623): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cc483552d457459499f9094afe4081d9: Network error: Client connection negotiation failed: client connection to 127.9.155.65:43623: connect: Connection refused (error 111)
I20260620 19:10:08.791611 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 11923
I20260620 19:10:08.792531 12218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:08.791795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.66:40669 (local address 127.0.0.1:59448)
0620 19:10:08.792393 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:08.792446 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.66:40669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":506}
I20260620 19:10:08.798323 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 11539
2026-06-20T19:10:08Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6410 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:10:08 dist-test-slave-n3fx krb5kdc[12221](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:10:08 dist-test-slave-n3fx krb5kdc[12221](info): set up 2 sockets
Jun 20 19:10:08 dist-test-slave-n3fx krb5kdc[12221](info): commencing operation
krb5kdc: starting...
W20260620 19:10:09.768289 9837 mini_kdc.cc:121] Time spent starting KDC: real 0.948s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/krb5kdc/test-user.keytab.
Jun 20 19:10:09 dist-test-slave-n3fx krb5kdc[12221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982609, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:10:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:10:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.155.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.155.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.155.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab.
Entry for principal kudu/127.9.155.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab.
I20260620 19:10:09.891639 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:43013
--webserver_interface=127.9.155.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab
--principal=kudu/127.9.155.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:42927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.155.126:43013
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:10:10.005488 12234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:10.005826 12234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:10.005872 12234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:10.011195 12234 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:10:10.011310 12234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:10.011333 12234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:10.011350 12234 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:10:10.011366 12234 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:10:10.018395 12234 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:42927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.155.126:43013
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab
--principal=kudu/127.9.155.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:43013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.155.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12234
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:10:10.019865 12234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:10.021183 12234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 19:10:10.031194 12234 server_base.cc:1034] running on GCE node
W20260620 19:10:10.031147 12242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:10.031149 12240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:10.031147 12239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:10.032074 12234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:10.033282 12234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:10.034476 12234 hybrid_clock.cc:648] HybridClock initialized: now 1781982610034451 us; error 41 us; skew 500 ppm
Jun 20 19:10:10 dist-test-slave-n3fx krb5kdc[12221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982610, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:10.037994 12234 init.cc:377] Logged in from keytab as kudu/127.9.155.126@KRBTEST.COM (short username kudu)
I20260620 19:10:10.039453 12234 webserver.cc:460] Webserver started at http://127.9.155.126:45239/ using document root <none> and password file <none>
I20260620 19:10:10.040182 12234 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:10.040323 12234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:10.040550 12234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:10:10.042868 12234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/minicluster-data/master-0/data/instance:
uuid: "f8ddc73f1ed943048c463894bb1659b6"
format_stamp: "Formatted at 2026-06-20 19:10:10 on dist-test-slave-n3fx"
server_key: "5accfb03d9a97c36e929cfad98572737"
server_key_iv: "2351f097394ca572a8d715d07f05cf18"
server_key_version: "encryptionkey@0"
I20260620 19:10:10.043489 12234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/minicluster-data/master-0/wal/instance:
uuid: "f8ddc73f1ed943048c463894bb1659b6"
format_stamp: "Formatted at 2026-06-20 19:10:10 on dist-test-slave-n3fx"
server_key: "5accfb03d9a97c36e929cfad98572737"
server_key_iv: "2351f097394ca572a8d715d07f05cf18"
server_key_version: "encryptionkey@0"
I20260620 19:10:10.047724 12234 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 19:10:10.050630 12249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:10.052129 12234 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 19:10:10.052311 12234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/minicluster-data/master-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/minicluster-data/master-0/wal
uuid: "f8ddc73f1ed943048c463894bb1659b6"
format_stamp: "Formatted at 2026-06-20 19:10:10 on dist-test-slave-n3fx"
server_key: "5accfb03d9a97c36e929cfad98572737"
server_key_iv: "2351f097394ca572a8d715d07f05cf18"
server_key_version: "encryptionkey@0"
I20260620 19:10:10.052453 12234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:10.070680 12234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:10.074520 12234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:10.074726 12234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:10.082197 12234 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.126:43013
I20260620 19:10:10.082247 12301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.126:43013 every 8 connection(s)
I20260620 19:10:10.083186 12234 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
I20260620 19:10:10.086081 12302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:10.088547 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 12234
I20260620 19:10:10.088658 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/minicluster-data/master-0/wal/instance
I20260620 19:10:10.089062 9837 external_mini_cluster.cc:1428] Setting key 70e6d129f383561cc303e587b27d0d1d
I20260620 19:10:10.092478 12302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8ddc73f1ed943048c463894bb1659b6: Bootstrap starting.
I20260620 19:10:10.094817 12302 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f8ddc73f1ed943048c463894bb1659b6: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:10.095589 12302 log.cc:824] T 00000000000000000000000000000000 P f8ddc73f1ed943048c463894bb1659b6: Log is configured to *not* fsync() on all Append() calls
Jun 20 19:10:10 dist-test-slave-n3fx krb5kdc[12221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982609, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:10:10.098951 12302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8ddc73f1ed943048c463894bb1659b6: No bootstrap required, opened a new log
I20260620 19:10:10.102381 12302 raft_consensus.cc:348] T 00000000000000000000000000000000 P f8ddc73f1ed943048c463894bb1659b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8ddc73f1ed943048c463894bb1659b6" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 43013 } }
I20260620 19:10:10.102641 12302 raft_consensus.cc:374] T 00000000000000000000000000000000 P f8ddc73f1ed943048c463894bb1659b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:10.102720 12302 raft_consensus.cc:729] T 00000000000000000000000000000000 P f8ddc73f1ed943048c463894bb1659b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8ddc73f1ed943048c463894bb1659b6, State: Initialized, Role: FOLLOWER
I20260620 19:10:10.103335 12302 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8ddc73f1ed943048c463894bb1659b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8ddc73f1ed943048c463894bb1659b6" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 43013 } }
I20260620 19:10:10.103483 12302 raft_consensus.cc:388] T 00000000000000000000000000000000 P f8ddc73f1ed943048c463894bb1659b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:10:10.103554 12302 raft_consensus.cc:482] T 00000000000000000000000000000000 P f8ddc73f1ed943048c463894bb1659b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:10:10.103655 12302 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f8ddc73f1ed943048c463894bb1659b6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:10.104852 12302 raft_consensus.cc:504] T 00000000000000000000000000000000 P f8ddc73f1ed943048c463894bb1659b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8ddc73f1ed943048c463894bb1659b6" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 43013 } }
I20260620 19:10:10.105207 12302 leader_election.cc:302] T 00000000000000000000000000000000 P f8ddc73f1ed943048c463894bb1659b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f8ddc73f1ed943048c463894bb1659b6; no voters:
I20260620 19:10:10.105525 12302 leader_election.cc:288] T 00000000000000000000000000000000 P f8ddc73f1ed943048c463894bb1659b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:10:10.105509 12305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:10.090599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52954 (local address 127.9.155.126:43013)
0620 19:10:10.091149 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:10.091166 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:10.091206 (+ 40us) server_negotiation.cc:407] Connection header received
0620 19:10:10.091860 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:10.091889 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:10.092311 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:10.092675 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:10.093508 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:10.094531 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:10.095524 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:10.096195 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:10.096495 (+ 300us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:10.099466 (+ 2971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:10.099472 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:10.099487 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:10.099518 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:10.102151 (+ 2633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:10.102627 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:10.102633 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:10.102637 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:10.102791 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:10.103185 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:10.103188 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:10.103189 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:10.103687 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:10.103871 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:10.104264 (+ 393us) server_negotiation.cc:299] Negotiation successful
0620 19:10:10.104618 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":148,"threads_started":1}
I20260620 19:10:10.105657 12307 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f8ddc73f1ed943048c463894bb1659b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:10:10.105916 12307 raft_consensus.cc:686] T 00000000000000000000000000000000 P f8ddc73f1ed943048c463894bb1659b6 [term 1 LEADER]: Becoming Leader. State: Replica: f8ddc73f1ed943048c463894bb1659b6, State: Running, Role: LEADER
I20260620 19:10:10.106281 12307 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8ddc73f1ed943048c463894bb1659b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8ddc73f1ed943048c463894bb1659b6" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 43013 } }
I20260620 19:10:10.106810 12302 sys_catalog.cc:564] T 00000000000000000000000000000000 P f8ddc73f1ed943048c463894bb1659b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:10:10.107547 12309 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8ddc73f1ed943048c463894bb1659b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f8ddc73f1ed943048c463894bb1659b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8ddc73f1ed943048c463894bb1659b6" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 43013 } } }
I20260620 19:10:10.107693 12309 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8ddc73f1ed943048c463894bb1659b6 [sys.catalog]: This master's current role is: LEADER
I20260620 19:10:10.107547 12307 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8ddc73f1ed943048c463894bb1659b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8ddc73f1ed943048c463894bb1659b6. Latest consensus state: current_term: 1 leader_uuid: "f8ddc73f1ed943048c463894bb1659b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8ddc73f1ed943048c463894bb1659b6" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 43013 } } }
I20260620 19:10:10.107970 12307 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8ddc73f1ed943048c463894bb1659b6 [sys.catalog]: This master's current role is: LEADER
I20260620 19:10:10.108594 12315 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:10:10.111766 12315 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:10:10.118251 12315 catalog_manager.cc:1347] Generated new cluster ID: 6c65c8c3bc3d4c15ac0eecba160f3912
I20260620 19:10:10.118381 12315 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:10:10.133816 12315 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:10:10.135195 12315 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:10:10.144145 12315 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f8ddc73f1ed943048c463894bb1659b6: Generated new TSK 0
I20260620 19:10:10.145087 12315 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.155.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.155.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.155.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.65.keytab.
Entry for principal kudu/127.9.155.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.65.keytab.
I20260620 19:10:10.179121 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.65:0
--local_ip_for_outbound_sockets=127.9.155.65
--webserver_interface=127.9.155.65
--webserver_port=0
--tserver_master_addrs=127.9.155.126:43013
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.65.keytab
--principal=kudu/127.9.155.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:42927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:10:10.294389 12328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:10.294708 12328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:10.294770 12328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:10.299018 12328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:10.299099 12328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:10.299186 12328 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.65
I20260620 19:10:10.304881 12328 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:42927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.65.keytab
--principal=kudu/127.9.155.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.155.65
--webserver_port=0
--tserver_master_addrs=127.9.155.126:43013
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12328
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.65
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:10:10.305995 12328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:10.307041 12328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:10:10.317651 12333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:10.317651 12336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:10.317626 12334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:10.318046 12328 server_base.cc:1034] running on GCE node
I20260620 19:10:10.318558 12328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:10.319335 12328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:10.320458 12328 hybrid_clock.cc:648] HybridClock initialized: now 1781982610320423 us; error 51 us; skew 500 ppm
Jun 20 19:10:10 dist-test-slave-n3fx krb5kdc[12221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982610, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:10.324021 12328 init.cc:377] Logged in from keytab as kudu/127.9.155.65@KRBTEST.COM (short username kudu)
I20260620 19:10:10.324955 12328 webserver.cc:460] Webserver started at http://127.9.155.65:34835/ using document root <none> and password file <none>
I20260620 19:10:10.325506 12328 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:10.325567 12328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:10.325755 12328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:10:10.328004 12328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-0/data/instance:
uuid: "39161e93f2f84aa0be4582468b5f1024"
format_stamp: "Formatted at 2026-06-20 19:10:10 on dist-test-slave-n3fx"
server_key: "36229d2bd1697f984b3151f5d0699c65"
server_key_iv: "d7017aff46edde3de0382a1493fa944d"
server_key_version: "encryptionkey@0"
I20260620 19:10:10.328560 12328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-0/wal/instance:
uuid: "39161e93f2f84aa0be4582468b5f1024"
format_stamp: "Formatted at 2026-06-20 19:10:10 on dist-test-slave-n3fx"
server_key: "36229d2bd1697f984b3151f5d0699c65"
server_key_iv: "d7017aff46edde3de0382a1493fa944d"
server_key_version: "encryptionkey@0"
I20260620 19:10:10.332259 12328 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:10:10.334798 12343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:10.336313 12328 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:10:10.336426 12328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-0/wal
uuid: "39161e93f2f84aa0be4582468b5f1024"
format_stamp: "Formatted at 2026-06-20 19:10:10 on dist-test-slave-n3fx"
server_key: "36229d2bd1697f984b3151f5d0699c65"
server_key_iv: "d7017aff46edde3de0382a1493fa944d"
server_key_version: "encryptionkey@0"
I20260620 19:10:10.336529 12328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:10.353430 12328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:10.356765 12328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:10.356923 12328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:10.357559 12328 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:10:10.358392 12328 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:10:10.358462 12328 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:10.358520 12328 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:10:10.358537 12328 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:10.367185 12328 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.65:34503
I20260620 19:10:10.367208 12456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.65:34503 every 8 connection(s)
I20260620 19:10:10.368018 12328 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
I20260620 19:10:10.376132 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 12328
I20260620 19:10:10.376269 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-0/wal/instance
I20260620 19:10:10.376569 9837 external_mini_cluster.cc:1428] Setting key 1c08b701fb4355b2611b7bdffa43b64f
Jun 20 19:10:10 dist-test-slave-n3fx krb5kdc[12221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982610, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.65@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:10:10.381153 12305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:10.369727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.65:41355 (local address 127.9.155.126:43013)
0620 19:10:10.369851 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:10.369854 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:10.370828 (+ 974us) server_negotiation.cc:407] Connection header received
0620 19:10:10.371724 (+ 896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:10.371728 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:10.371783 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:10.371856 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:10.373056 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:10.373592 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:10.374686 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:10.374917 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:10.375004 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:10.377624 (+ 2620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:10.377628 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:10.377631 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:10.377656 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:10.379000 (+ 1344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:10.379538 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:10.379541 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:10.379544 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:10.379633 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:10.380021 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:10.380024 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:10.380025 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:10.380264 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:10.380356 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:10.380870 (+ 514us) server_negotiation.cc:299] Negotiation successful
0620 19:10:10.380996 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 19:10:10.382071 12459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:10.370059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:43013 (local address 127.9.155.65:41355)
0620 19:10:10.370686 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:10.370726 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:10.371520 (+ 794us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:10.372041 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:10.372047 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:10.372436 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:10.372914 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:10.372928 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:10.373709 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:10.373712 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:10.374575 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:10.374584 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:10.375056 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:10.375060 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:10.375209 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:10.375793 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:10.375810 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:10.377475 (+ 1665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:10.379110 (+ 1635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:10.379117 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:10.379135 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:10.379450 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:10.379756 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:10.379759 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:10.379766 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:10.379927 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:10.380359 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:10.380365 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:10.380624 (+ 259us) client_negotiation.cc:769] Sending connection context
0620 19:10:10.380812 (+ 188us) client_negotiation.cc:240] Negotiation successful
0620 19:10:10.381092 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":112,"threads_started":1}
I20260620 19:10:10.383416 12457 heartbeater.cc:347] Connected to a master server at 127.9.155.126:43013
I20260620 19:10:10.383694 12457 heartbeater.cc:464] Registering TS with master...
I20260620 19:10:10.384394 12457 heartbeater.cc:511] Master 127.9.155.126:43013 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.9.155.66@KRBTEST.COM; defaulting to no policy
I20260620 19:10:10.386150 12266 ts_manager.cc:194] Registered new tserver with Master: 39161e93f2f84aa0be4582468b5f1024 (127.9.155.65:34503)
I20260620 19:10:10.387353 12266 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.155.65@KRBTEST.COM'} at 127.9.155.65:41355
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.155.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.155.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.66.keytab.
Entry for principal kudu/127.9.155.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.66.keytab.
I20260620 19:10:10.403842 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.66:0
--local_ip_for_outbound_sockets=127.9.155.66
--webserver_interface=127.9.155.66
--webserver_port=0
--tserver_master_addrs=127.9.155.126:43013
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.66.keytab
--principal=kudu/127.9.155.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:42927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:10:10.512462 12462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:10.512734 12462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:10.512784 12462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:10.516884 12462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:10.516954 12462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:10.517014 12462 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.66
I20260620 19:10:10.522429 12462 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:42927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.66.keytab
--principal=kudu/127.9.155.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.155.66
--webserver_port=0
--tserver_master_addrs=127.9.155.126:43013
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12462
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.66
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:10:10.523519 12462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:10.524583 12462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:10:10.534312 12468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:10.534404 12467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:10.534318 12470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:10.534579 12462 server_base.cc:1034] running on GCE node
I20260620 19:10:10.535082 12462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:10.535818 12462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:10.536993 12462 hybrid_clock.cc:648] HybridClock initialized: now 1781982610536963 us; error 47 us; skew 500 ppm
Jun 20 19:10:10 dist-test-slave-n3fx krb5kdc[12221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982610, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:10.539759 12462 init.cc:377] Logged in from keytab as kudu/127.9.155.66@KRBTEST.COM (short username kudu)
I20260620 19:10:10.540678 12462 webserver.cc:460] Webserver started at http://127.9.155.66:35327/ using document root <none> and password file <none>
I20260620 19:10:10.541214 12462 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:10.541275 12462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:10.541479 12462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:10:10.543623 12462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-1/data/instance:
uuid: "4de4461a34b4457c8b21a2ce598fc12b"
format_stamp: "Formatted at 2026-06-20 19:10:10 on dist-test-slave-n3fx"
server_key: "c77f1a48b39bb0b7192f83dbc7bbc866"
server_key_iv: "cd662afbc66130ac28c7af94f8978650"
server_key_version: "encryptionkey@0"
I20260620 19:10:10.544389 12462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-1/wal/instance:
uuid: "4de4461a34b4457c8b21a2ce598fc12b"
format_stamp: "Formatted at 2026-06-20 19:10:10 on dist-test-slave-n3fx"
server_key: "c77f1a48b39bb0b7192f83dbc7bbc866"
server_key_iv: "cd662afbc66130ac28c7af94f8978650"
server_key_version: "encryptionkey@0"
I20260620 19:10:10.548346 12462 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 19:10:10.550921 12477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:10.552356 12462 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:10:10.552475 12462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-1/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-1/wal
uuid: "4de4461a34b4457c8b21a2ce598fc12b"
format_stamp: "Formatted at 2026-06-20 19:10:10 on dist-test-slave-n3fx"
server_key: "c77f1a48b39bb0b7192f83dbc7bbc866"
server_key_iv: "cd662afbc66130ac28c7af94f8978650"
server_key_version: "encryptionkey@0"
I20260620 19:10:10.552577 12462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:10.573406 12462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:10.576340 12462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:10.576498 12462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:10.577047 12462 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:10:10.577934 12462 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:10:10.577980 12462 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:10.578019 12462 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:10:10.578033 12462 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:10.586431 12462 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.66:35551
I20260620 19:10:10.586462 12590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.66:35551 every 8 connection(s)
I20260620 19:10:10.587304 12462 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
I20260620 19:10:10.590215 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 12462
I20260620 19:10:10.590325 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-1/wal/instance
I20260620 19:10:10.590617 9837 external_mini_cluster.cc:1428] Setting key ed55306299b19a9d3305a9f1ed91e24c
Jun 20 19:10:10 dist-test-slave-n3fx krb5kdc[12221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982610, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.66@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.155.67@KRBTEST.COM; defaulting to no policy
I20260620 19:10:10.601323 12305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:10.589097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.66:40837 (local address 127.9.155.126:43013)
0620 19:10:10.589260 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:10.589265 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:10.590324 (+ 1059us) server_negotiation.cc:407] Connection header received
0620 19:10:10.591421 (+ 1097us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:10.591423 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:10.591466 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:10.591530 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:10.593061 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:10.593676 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:10.594882 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:10.595128 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:10.595225 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:10.597849 (+ 2624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:10.597852 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:10.597854 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:10.597878 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:10.599138 (+ 1260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:10.599727 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:10.599731 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:10.599733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:10.599820 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:10.600105 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:10.600108 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:10.600109 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:10.600367 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:10.600460 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:10.600959 (+ 499us) server_negotiation.cc:299] Negotiation successful
0620 19:10:10.601071 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.155.67@KRBTEST.COM" created.
I20260620 19:10:10.602062 12593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:10.589455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:43013 (local address 127.9.155.66:40837)
0620 19:10:10.590152 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:10.590211 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:10.591217 (+ 1006us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:10.591724 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:10.591733 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:10.592160 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:10.592847 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:10.592867 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:10.593813 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:10.593816 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:10.594748 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:10.594759 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:10.595292 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:10.595297 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:10.595455 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:10.596008 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:10.596024 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:10.597662 (+ 1638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:10.599254 (+ 1592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:10.599261 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:10.599277 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:10.599627 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:10.599901 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:10.599903 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:10.599905 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:10.600040 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:10.600457 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:10.600462 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:10.600711 (+ 249us) client_negotiation.cc:769] Sending connection context
0620 19:10:10.600884 (+ 173us) client_negotiation.cc:240] Negotiation successful
0620 19:10:10.601146 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":112,"threads_started":1}
I20260620 19:10:10.603308 12591 heartbeater.cc:347] Connected to a master server at 127.9.155.126:43013
I20260620 19:10:10.603574 12591 heartbeater.cc:464] Registering TS with master...
I20260620 19:10:10.604156 12591 heartbeater.cc:511] Master 127.9.155.126:43013 requested a full tablet report, sending...
I20260620 19:10:10.605440 12266 ts_manager.cc:194] Registered new tserver with Master: 4de4461a34b4457c8b21a2ce598fc12b (127.9.155.66:35551)
I20260620 19:10:10.606179 12266 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.155.66@KRBTEST.COM'} at 127.9.155.66:40837
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.155.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.67.keytab.
Entry for principal kudu/127.9.155.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.67.keytab.
I20260620 19:10:10.618690 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.67:0
--local_ip_for_outbound_sockets=127.9.155.67
--webserver_interface=127.9.155.67
--webserver_port=0
--tserver_master_addrs=127.9.155.126:43013
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.67.keytab
--principal=kudu/127.9.155.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:42927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:10:10.728996 12596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:10.729293 12596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:10.729338 12596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:10.733687 12596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:10.733759 12596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:10.733821 12596 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.67
I20260620 19:10:10.739245 12596 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:42927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.67.keytab
--principal=kudu/127.9.155.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.155.67
--webserver_port=0
--tserver_master_addrs=127.9.155.126:43013
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12596
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.67
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:10:10.740377 12596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:10.741425 12596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:10:10.750991 12604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:10.751017 12602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:10.751017 12601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:10.751400 12596 server_base.cc:1034] running on GCE node
I20260620 19:10:10.751829 12596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:10.752514 12596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:10.753676 12596 hybrid_clock.cc:648] HybridClock initialized: now 1781982610753640 us; error 50 us; skew 500 ppm
Jun 20 19:10:10 dist-test-slave-n3fx krb5kdc[12221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982610, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:10.756564 12596 init.cc:377] Logged in from keytab as kudu/127.9.155.67@KRBTEST.COM (short username kudu)
I20260620 19:10:10.757484 12596 webserver.cc:460] Webserver started at http://127.9.155.67:45875/ using document root <none> and password file <none>
I20260620 19:10:10.758085 12596 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:10.758153 12596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:10.758356 12596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:10:10.760473 12596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-2/data/instance:
uuid: "ad2708b8586b42c8939a55011435ca00"
format_stamp: "Formatted at 2026-06-20 19:10:10 on dist-test-slave-n3fx"
server_key: "29880adf74ba0c4bb804305012cfbe73"
server_key_iv: "3cee28c9d7f6b2476bb7c783491baf64"
server_key_version: "encryptionkey@0"
I20260620 19:10:10.761005 12596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-2/wal/instance:
uuid: "ad2708b8586b42c8939a55011435ca00"
format_stamp: "Formatted at 2026-06-20 19:10:10 on dist-test-slave-n3fx"
server_key: "29880adf74ba0c4bb804305012cfbe73"
server_key_iv: "3cee28c9d7f6b2476bb7c783491baf64"
server_key_version: "encryptionkey@0"
I20260620 19:10:10.764721 12596 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 19:10:10.767192 12611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:10.768584 12596 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:10:10.768683 12596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-2/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-2/wal
uuid: "ad2708b8586b42c8939a55011435ca00"
format_stamp: "Formatted at 2026-06-20 19:10:10 on dist-test-slave-n3fx"
server_key: "29880adf74ba0c4bb804305012cfbe73"
server_key_iv: "3cee28c9d7f6b2476bb7c783491baf64"
server_key_version: "encryptionkey@0"
I20260620 19:10:10.768791 12596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:10.824529 12596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:10.827955 12596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:10.828145 12596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:10.828809 12596 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:10:10.829643 12596 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:10:10.829689 12596 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:10.829733 12596 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:10:10.829763 12596 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:10.838327 12596 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.67:40199
I20260620 19:10:10.838346 12724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.67:40199 every 8 connection(s)
I20260620 19:10:10.839260 12596 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
I20260620 19:10:10.846169 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 12596
I20260620 19:10:10.846302 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-2/wal/instance
I20260620 19:10:10.846549 9837 external_mini_cluster.cc:1428] Setting key 03a220f55e902661922e1a7a38e59459
Jun 20 19:10:10 dist-test-slave-n3fx krb5kdc[12221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982610, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.67@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:10:10.853015 12305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:10.841160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.67:35889 (local address 127.9.155.126:43013)
0620 19:10:10.841340 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:10.841344 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:10.842323 (+ 979us) server_negotiation.cc:407] Connection header received
0620 19:10:10.843200 (+ 877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:10.843203 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:10.843260 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:10.843328 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:10.844530 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:10.845064 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:10.846376 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:10.846634 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:10.846731 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:10.849225 (+ 2494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:10.849228 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:10.849230 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:10.849256 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:10.850690 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:10.851247 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:10.851251 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:10.851254 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:10.851335 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:10.851771 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:10.851775 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:10.851776 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:10.851982 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:10.852070 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:10.852756 (+ 686us) server_negotiation.cc:299] Negotiation successful
0620 19:10:10.852889 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260620 19:10:10.853951 12727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:10.841496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:43013 (local address 127.9.155.67:35889)
0620 19:10:10.842157 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:10.842198 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:10.843005 (+ 807us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:10.843497 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:10.843503 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:10.843859 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:10.844389 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:10.844407 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:10.845188 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:10.845191 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:10.846222 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:10.846244 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:10.846793 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:10.846797 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:10.846964 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:10.847534 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:10.847554 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:10.849072 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:10.850834 (+ 1762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:10.850840 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:10.850854 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:10.851126 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:10.851457 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:10.851459 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:10.851461 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:10.851623 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:10.852102 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:10.852108 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:10.852468 (+ 360us) client_negotiation.cc:769] Sending connection context
0620 19:10:10.852674 (+ 206us) client_negotiation.cc:240] Negotiation successful
0620 19:10:10.853085 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":111,"threads_started":1}
I20260620 19:10:10.855149 12725 heartbeater.cc:347] Connected to a master server at 127.9.155.126:43013
I20260620 19:10:10.855461 12725 heartbeater.cc:464] Registering TS with master...
I20260620 19:10:10.856009 12725 heartbeater.cc:511] Master 127.9.155.126:43013 requested a full tablet report, sending...
I20260620 19:10:10.857256 12266 ts_manager.cc:194] Registered new tserver with Master: ad2708b8586b42c8939a55011435ca00 (127.9.155.67:40199)
I20260620 19:10:10.857848 12266 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.155.67@KRBTEST.COM'} at 127.9.155.67:35889
I20260620 19:10:10.860673 9837 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 19:10:10.873206 12305 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:10:10.871611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52962 (local address 127.9.155.126:43013)
0620 19:10:10.871814 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:10.871818 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:10.871908 (+ 90us) server_negotiation.cc:407] Connection header received
0620 19:10:10.872012 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:10.872014 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:10.872063 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:10.872127 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:10.873070 (+ 943us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52962: BlockingRecv error: recv got EOF from 127.0.0.1:52962 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 19:10:10.874302 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 12328
I20260620 19:10:10.879068 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 12462
I20260620 19:10:10.884065 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 12596
I20260620 19:10:10.889379 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 12234
2026-06-20T19:10:10Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2089 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982595231230-9837-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:10:10 dist-test-slave-n3fx krb5kdc[12737](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:10:10 dist-test-slave-n3fx krb5kdc[12737](info): set up 2 sockets
Jun 20 19:10:10 dist-test-slave-n3fx krb5kdc[12737](info): commencing operation
krb5kdc: starting...
W20260620 19:10:11.843940 9837 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982595231230-9837-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982595231230-9837-0/krb5kdc/test-user.keytab.
Jun 20 19:10:11 dist-test-slave-n3fx krb5kdc[12737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982611, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:10:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:10:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.155.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.155.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.155.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab.
Entry for principal kudu/127.9.155.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab.
I20260620 19:10:11.964962 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982595231230-9837-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982595231230-9837-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982595231230-9837-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:37851
--webserver_interface=127.9.155.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab
--principal=kudu/127.9.155.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:32851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.155.126:37851
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:10:12.074885 12750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:12.075198 12750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:12.075393 12750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:12.079941 12750 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:10:12.080036 12750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:12.080070 12750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:12.080103 12750 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:10:12.080124 12750 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:10:12.085875 12750 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:32851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982595231230-9837-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982595231230-9837-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.155.126:37851
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab
--principal=kudu/127.9.155.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:37851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.155.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12750
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982595231230-9837-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:10:12.087239 12750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:12.088510 12750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:10:12.097522 12758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:12.097549 12756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:12.097522 12755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:12.097878 12750 server_base.cc:1034] running on GCE node
I20260620 19:10:12.098429 12750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:12.099692 12750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:12.100878 12750 hybrid_clock.cc:648] HybridClock initialized: now 1781982612100839 us; error 54 us; skew 500 ppm
Jun 20 19:10:12 dist-test-slave-n3fx krb5kdc[12737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982612, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:12.104120 12750 init.cc:377] Logged in from keytab as kudu/127.9.155.126@KRBTEST.COM (short username kudu)
I20260620 19:10:12.105167 12750 webserver.cc:460] Webserver started at http://127.9.155.126:35121/ using document root <none> and password file <none>
I20260620 19:10:12.105729 12750 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:12.105798 12750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:12.105978 12750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:10:12.108209 12750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982595231230-9837-0/minicluster-data/master-0/data/instance:
uuid: "e996029b8a714122bca1386fcfe59515"
format_stamp: "Formatted at 2026-06-20 19:10:12 on dist-test-slave-n3fx"
server_key: "f95b48e0a525ff1626234d2ec2f24820"
server_key_iv: "edf9da3a1441196261e8cf55d676e880"
server_key_version: "encryptionkey@0"
I20260620 19:10:12.108810 12750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982595231230-9837-0/minicluster-data/master-0/wal/instance:
uuid: "e996029b8a714122bca1386fcfe59515"
format_stamp: "Formatted at 2026-06-20 19:10:12 on dist-test-slave-n3fx"
server_key: "f95b48e0a525ff1626234d2ec2f24820"
server_key_iv: "edf9da3a1441196261e8cf55d676e880"
server_key_version: "encryptionkey@0"
I20260620 19:10:12.112607 12750 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 19:10:12.115170 12765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:12.116544 12750 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:10:12.116652 12750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982595231230-9837-0/minicluster-data/master-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982595231230-9837-0/minicluster-data/master-0/wal
uuid: "e996029b8a714122bca1386fcfe59515"
format_stamp: "Formatted at 2026-06-20 19:10:12 on dist-test-slave-n3fx"
server_key: "f95b48e0a525ff1626234d2ec2f24820"
server_key_iv: "edf9da3a1441196261e8cf55d676e880"
server_key_version: "encryptionkey@0"
I20260620 19:10:12.116746 12750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982595231230-9837-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982595231230-9837-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982595231230-9837-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:12.136713 12750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:12.140007 12750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:12.140170 12750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:12.147187 12750 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.126:37851
I20260620 19:10:12.147485 12817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.126:37851 every 8 connection(s)
I20260620 19:10:12.148164 12750 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
I20260620 19:10:12.151332 12818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:12.151593 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 12750
I20260620 19:10:12.151691 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982595231230-9837-0/minicluster-data/master-0/wal/instance
I20260620 19:10:12.151988 9837 external_mini_cluster.cc:1428] Setting key d37162ca8f0fd53c0c096704e8d8620a
I20260620 19:10:12.158434 12818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e996029b8a714122bca1386fcfe59515: Bootstrap starting.
I20260620 19:10:12.161114 12818 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e996029b8a714122bca1386fcfe59515: Neither blocks nor log segments found. Creating new log.
Jun 20 19:10:12 dist-test-slave-n3fx krb5kdc[12737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982611, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:10:12.161864 12818 log.cc:824] T 00000000000000000000000000000000 P e996029b8a714122bca1386fcfe59515: Log is configured to *not* fsync() on all Append() calls
I20260620 19:10:12.163784 12818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e996029b8a714122bca1386fcfe59515: No bootstrap required, opened a new log
I20260620 19:10:12.166368 12821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:12.153618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44648 (local address 127.9.155.126:37851)
0620 19:10:12.154053 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:12.154069 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:12.154105 (+ 36us) server_negotiation.cc:407] Connection header received
0620 19:10:12.154772 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:12.154799 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:12.155142 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:12.155429 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:12.156429 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:12.157475 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:12.158653 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:12.159162 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:12.159303 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:12.162040 (+ 2737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:12.162047 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:12.162065 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:12.162098 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:12.163655 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:12.164108 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:12.164114 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:12.164118 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:12.164255 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:12.164510 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:12.164513 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:12.164515 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:12.164907 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:12.165091 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:12.165350 (+ 259us) server_negotiation.cc:299] Negotiation successful
0620 19:10:12.165631 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":116,"threads_started":1}
I20260620 19:10:12.167409 12818 raft_consensus.cc:348] T 00000000000000000000000000000000 P e996029b8a714122bca1386fcfe59515 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e996029b8a714122bca1386fcfe59515" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 37851 } }
I20260620 19:10:12.167639 12818 raft_consensus.cc:374] T 00000000000000000000000000000000 P e996029b8a714122bca1386fcfe59515 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:12.167689 12818 raft_consensus.cc:729] T 00000000000000000000000000000000 P e996029b8a714122bca1386fcfe59515 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e996029b8a714122bca1386fcfe59515, State: Initialized, Role: FOLLOWER
I20260620 19:10:12.168126 12818 consensus_queue.cc:260] T 00000000000000000000000000000000 P e996029b8a714122bca1386fcfe59515 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e996029b8a714122bca1386fcfe59515" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 37851 } }
I20260620 19:10:12.168313 12818 raft_consensus.cc:388] T 00000000000000000000000000000000 P e996029b8a714122bca1386fcfe59515 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:10:12.168387 12818 raft_consensus.cc:482] T 00000000000000000000000000000000 P e996029b8a714122bca1386fcfe59515 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:10:12.168485 12818 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e996029b8a714122bca1386fcfe59515 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:12.169312 9837 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:10:12.169504 12818 raft_consensus.cc:504] T 00000000000000000000000000000000 P e996029b8a714122bca1386fcfe59515 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e996029b8a714122bca1386fcfe59515" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 37851 } }
I20260620 19:10:12.169840 12818 leader_election.cc:302] T 00000000000000000000000000000000 P e996029b8a714122bca1386fcfe59515 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e996029b8a714122bca1386fcfe59515; no voters:
I20260620 19:10:12.170138 12818 leader_election.cc:288] T 00000000000000000000000000000000 P e996029b8a714122bca1386fcfe59515 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:10:12.170255 12823 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e996029b8a714122bca1386fcfe59515 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:10:12.170475 12823 raft_consensus.cc:686] T 00000000000000000000000000000000 P e996029b8a714122bca1386fcfe59515 [term 1 LEADER]: Becoming Leader. State: Replica: e996029b8a714122bca1386fcfe59515, State: Running, Role: LEADER
I20260620 19:10:12.170812 12823 consensus_queue.cc:237] T 00000000000000000000000000000000 P e996029b8a714122bca1386fcfe59515 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e996029b8a714122bca1386fcfe59515" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 37851 } }
I20260620 19:10:12.171588 12818 sys_catalog.cc:564] T 00000000000000000000000000000000 P e996029b8a714122bca1386fcfe59515 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:10:12.172556 12824 sys_catalog.cc:455] T 00000000000000000000000000000000 P e996029b8a714122bca1386fcfe59515 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e996029b8a714122bca1386fcfe59515" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e996029b8a714122bca1386fcfe59515" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 37851 } } }
I20260620 19:10:12.172688 12824 sys_catalog.cc:458] T 00000000000000000000000000000000 P e996029b8a714122bca1386fcfe59515 [sys.catalog]: This master's current role is: LEADER
I20260620 19:10:12.172922 12824 sys_catalog.cc:455] T 00000000000000000000000000000000 P e996029b8a714122bca1386fcfe59515 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e996029b8a714122bca1386fcfe59515. Latest consensus state: current_term: 1 leader_uuid: "e996029b8a714122bca1386fcfe59515" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e996029b8a714122bca1386fcfe59515" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 37851 } } }
I20260620 19:10:12.173026 12824 sys_catalog.cc:458] T 00000000000000000000000000000000 P e996029b8a714122bca1386fcfe59515 [sys.catalog]: This master's current role is: LEADER
W20260620 19:10:12.177196 12839 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e996029b8a714122bca1386fcfe59515: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:10:12.177281 12839 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:10:12.177364 12833 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:10:12.178056 12833 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:10:12.184096 12833 catalog_manager.cc:1347] Generated new cluster ID: 864f9ffad5b549bca92beea40b4bf8c9
I20260620 19:10:12.184171 12833 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:10:12.190598 12821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:12.185056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44662 (local address 127.9.155.126:37851)
0620 19:10:12.185169 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:12.185174 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:12.185465 (+ 291us) server_negotiation.cc:407] Connection header received
0620 19:10:12.185594 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:12.185596 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:12.185643 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:12.185719 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:12.186321 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:12.187084 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:12.188318 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:12.188681 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:12.188768 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:12.189648 (+ 880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:12.189652 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:12.189655 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:10:12.190172 (+ 517us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:12.190203 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:10:12.190221 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:12.190297 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:12.190397 (+ 100us) server_negotiation.cc:299] Negotiation successful
0620 19:10:12.190454 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 19:10:12.208706 12833 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:10:12.210319 12833 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:10:12.219851 12833 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e996029b8a714122bca1386fcfe59515: Generated new TSK 0
I20260620 19:10:12.221004 12833 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:10:12.229449 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 12750
2026-06-20T19:10:12Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1338 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:10:12 dist-test-slave-n3fx krb5kdc[12851](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:10:12 dist-test-slave-n3fx krb5kdc[12851](info): set up 2 sockets
Jun 20 19:10:12 dist-test-slave-n3fx krb5kdc[12851](info): commencing operation
krb5kdc: starting...
W20260620 19:10:13.207988 9837 mini_kdc.cc:121] Time spent starting KDC: real 0.959s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/krb5kdc/test-user.keytab.
Jun 20 19:10:13 dist-test-slave-n3fx krb5kdc[12851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:10:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:10:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.155.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.155.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.155.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab.
Entry for principal kudu/127.9.155.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab.
I20260620 19:10:13.328120 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:43851
--webserver_interface=127.9.155.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab
--principal=kudu/127.9.155.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:41489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.155.126:43851
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:10:13.439842 12864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:13.440178 12864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:13.440272 12864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:13.444656 12864 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:10:13.444721 12864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:13.444748 12864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:13.444769 12864 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:10:13.444797 12864 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:10:13.450219 12864 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:41489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.155.126:43851
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab
--principal=kudu/127.9.155.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:43851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.155.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12864
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:10:13.451325 12864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:13.452351 12864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 19:10:13.460884 12864 server_base.cc:1034] running on GCE node
W20260620 19:10:13.461078 12872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:13.461099 12869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:13.461082 12870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:13.461760 12864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:13.462893 12864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:13.464107 12864 hybrid_clock.cc:648] HybridClock initialized: now 1781982613464083 us; error 49 us; skew 500 ppm
Jun 20 19:10:13 dist-test-slave-n3fx krb5kdc[12851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982613, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:13.467036 12864 init.cc:377] Logged in from keytab as kudu/127.9.155.126@KRBTEST.COM (short username kudu)
I20260620 19:10:13.468019 12864 webserver.cc:460] Webserver started at http://127.9.155.126:38523/ using document root <none> and password file <none>
I20260620 19:10:13.468618 12864 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:13.468685 12864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:13.468892 12864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:10:13.470950 12864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/minicluster-data/master-0/data/instance:
uuid: "d61329ef0c4d44b28a0ffc3f56d7cfc7"
format_stamp: "Formatted at 2026-06-20 19:10:13 on dist-test-slave-n3fx"
server_key: "ec42195c5185d85bf78f1870637e7ea8"
server_key_iv: "d0b38faaeb2071960643cf0a5ee14f73"
server_key_version: "encryptionkey@0"
I20260620 19:10:13.471498 12864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/minicluster-data/master-0/wal/instance:
uuid: "d61329ef0c4d44b28a0ffc3f56d7cfc7"
format_stamp: "Formatted at 2026-06-20 19:10:13 on dist-test-slave-n3fx"
server_key: "ec42195c5185d85bf78f1870637e7ea8"
server_key_iv: "d0b38faaeb2071960643cf0a5ee14f73"
server_key_version: "encryptionkey@0"
I20260620 19:10:13.475358 12864 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 19:10:13.477854 12879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:13.479086 12864 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 19:10:13.479202 12864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/minicluster-data/master-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/minicluster-data/master-0/wal
uuid: "d61329ef0c4d44b28a0ffc3f56d7cfc7"
format_stamp: "Formatted at 2026-06-20 19:10:13 on dist-test-slave-n3fx"
server_key: "ec42195c5185d85bf78f1870637e7ea8"
server_key_iv: "d0b38faaeb2071960643cf0a5ee14f73"
server_key_version: "encryptionkey@0"
I20260620 19:10:13.479307 12864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:13.491767 12864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:13.494866 12864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:13.495049 12864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:13.501906 12864 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.126:43851
I20260620 19:10:13.502156 12931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.126:43851 every 8 connection(s)
I20260620 19:10:13.502861 12864 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
I20260620 19:10:13.504416 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 12864
I20260620 19:10:13.504518 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/minicluster-data/master-0/wal/instance
I20260620 19:10:13.504762 9837 external_mini_cluster.cc:1428] Setting key c66833767baff271dda5325a49545482
I20260620 19:10:13.505976 12932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:13.512391 12932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d61329ef0c4d44b28a0ffc3f56d7cfc7: Bootstrap starting.
Jun 20 19:10:13 dist-test-slave-n3fx krb5kdc[12851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:10:13.514966 12932 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d61329ef0c4d44b28a0ffc3f56d7cfc7: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:13.515687 12932 log.cc:824] T 00000000000000000000000000000000 P d61329ef0c4d44b28a0ffc3f56d7cfc7: Log is configured to *not* fsync() on all Append() calls
I20260620 19:10:13.517992 12932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d61329ef0c4d44b28a0ffc3f56d7cfc7: No bootstrap required, opened a new log
I20260620 19:10:13.519316 12935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:13.506054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56438 (local address 127.9.155.126:43851)
0620 19:10:13.506471 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:13.506490 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:13.506526 (+ 36us) server_negotiation.cc:407] Connection header received
0620 19:10:13.507140 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:13.507155 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:13.507474 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:13.507746 (+ 272us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:13.508674 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:13.509686 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:13.510839 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:13.511375 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:13.511497 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:13.514375 (+ 2878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:13.514381 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:13.514395 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:13.514426 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:13.516308 (+ 1882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:13.516828 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:13.516833 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:13.516838 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:13.516960 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:13.517275 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:13.517278 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:13.517280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:13.517693 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:13.517860 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:13.518232 (+ 372us) server_negotiation.cc:299] Negotiation successful
0620 19:10:13.518580 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":136,"threads_started":1}
I20260620 19:10:13.521860 12932 raft_consensus.cc:348] T 00000000000000000000000000000000 P d61329ef0c4d44b28a0ffc3f56d7cfc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d61329ef0c4d44b28a0ffc3f56d7cfc7" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 43851 } }
I20260620 19:10:13.522096 12932 raft_consensus.cc:374] T 00000000000000000000000000000000 P d61329ef0c4d44b28a0ffc3f56d7cfc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:13.522132 12932 raft_consensus.cc:729] T 00000000000000000000000000000000 P d61329ef0c4d44b28a0ffc3f56d7cfc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d61329ef0c4d44b28a0ffc3f56d7cfc7, State: Initialized, Role: FOLLOWER
I20260620 19:10:13.522567 12932 consensus_queue.cc:260] T 00000000000000000000000000000000 P d61329ef0c4d44b28a0ffc3f56d7cfc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d61329ef0c4d44b28a0ffc3f56d7cfc7" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 43851 } }
I20260620 19:10:13.522691 12932 raft_consensus.cc:388] T 00000000000000000000000000000000 P d61329ef0c4d44b28a0ffc3f56d7cfc7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:10:13.522751 12932 raft_consensus.cc:482] T 00000000000000000000000000000000 P d61329ef0c4d44b28a0ffc3f56d7cfc7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:10:13.522838 12932 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d61329ef0c4d44b28a0ffc3f56d7cfc7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:13.523800 12932 raft_consensus.cc:504] T 00000000000000000000000000000000 P d61329ef0c4d44b28a0ffc3f56d7cfc7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d61329ef0c4d44b28a0ffc3f56d7cfc7" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 43851 } }
I20260620 19:10:13.524130 12932 leader_election.cc:302] T 00000000000000000000000000000000 P d61329ef0c4d44b28a0ffc3f56d7cfc7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d61329ef0c4d44b28a0ffc3f56d7cfc7; no voters:
I20260620 19:10:13.524463 12932 leader_election.cc:288] T 00000000000000000000000000000000 P d61329ef0c4d44b28a0ffc3f56d7cfc7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:10:13.524590 12937 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d61329ef0c4d44b28a0ffc3f56d7cfc7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:10:13.524832 12937 raft_consensus.cc:686] T 00000000000000000000000000000000 P d61329ef0c4d44b28a0ffc3f56d7cfc7 [term 1 LEADER]: Becoming Leader. State: Replica: d61329ef0c4d44b28a0ffc3f56d7cfc7, State: Running, Role: LEADER
I20260620 19:10:13.525135 12937 consensus_queue.cc:237] T 00000000000000000000000000000000 P d61329ef0c4d44b28a0ffc3f56d7cfc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d61329ef0c4d44b28a0ffc3f56d7cfc7" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 43851 } }
I20260620 19:10:13.525749 12932 sys_catalog.cc:564] T 00000000000000000000000000000000 P d61329ef0c4d44b28a0ffc3f56d7cfc7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:10:13.526868 12939 sys_catalog.cc:455] T 00000000000000000000000000000000 P d61329ef0c4d44b28a0ffc3f56d7cfc7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d61329ef0c4d44b28a0ffc3f56d7cfc7. Latest consensus state: current_term: 1 leader_uuid: "d61329ef0c4d44b28a0ffc3f56d7cfc7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d61329ef0c4d44b28a0ffc3f56d7cfc7" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 43851 } } }
I20260620 19:10:13.527022 12939 sys_catalog.cc:458] T 00000000000000000000000000000000 P d61329ef0c4d44b28a0ffc3f56d7cfc7 [sys.catalog]: This master's current role is: LEADER
I20260620 19:10:13.526888 12938 sys_catalog.cc:455] T 00000000000000000000000000000000 P d61329ef0c4d44b28a0ffc3f56d7cfc7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d61329ef0c4d44b28a0ffc3f56d7cfc7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d61329ef0c4d44b28a0ffc3f56d7cfc7" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 43851 } } }
I20260620 19:10:13.527107 12938 sys_catalog.cc:458] T 00000000000000000000000000000000 P d61329ef0c4d44b28a0ffc3f56d7cfc7 [sys.catalog]: This master's current role is: LEADER
I20260620 19:10:13.527433 12946 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:10:13.530130 12946 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:10:13.536105 12946 catalog_manager.cc:1347] Generated new cluster ID: cf89895e161b4c129fd53da1bbd08670
I20260620 19:10:13.536194 12946 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:10:13.543533 12946 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:10:13.544940 12946 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:10:13.567617 12946 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d61329ef0c4d44b28a0ffc3f56d7cfc7: Generated new TSK 0
I20260620 19:10:13.568557 12946 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.155.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.155.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.155.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.65.keytab.
Entry for principal kudu/127.9.155.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.65.keytab.
I20260620 19:10:13.605194 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.65:0
--local_ip_for_outbound_sockets=127.9.155.65
--webserver_interface=127.9.155.65
--webserver_port=0
--tserver_master_addrs=127.9.155.126:43851
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.65.keytab
--principal=kudu/127.9.155.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:41489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:10:13.714345 12958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:13.714605 12958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:13.714648 12958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:13.718719 12958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:13.718786 12958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:13.718849 12958 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.65
I20260620 19:10:13.724174 12958 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:41489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.65.keytab
--principal=kudu/127.9.155.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.155.65
--webserver_port=0
--tserver_master_addrs=127.9.155.126:43851
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.65
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:10:13.725248 12958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:13.726302 12958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:10:13.736047 12964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:13.736047 12963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:13.736207 12958 server_base.cc:1034] running on GCE node
W20260620 19:10:13.736047 12966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:13.736856 12958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:13.737504 12958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:13.738682 12958 hybrid_clock.cc:648] HybridClock initialized: now 1781982613738666 us; error 32 us; skew 500 ppm
Jun 20 19:10:13 dist-test-slave-n3fx krb5kdc[12851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982613, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:13.741501 12958 init.cc:377] Logged in from keytab as kudu/127.9.155.65@KRBTEST.COM (short username kudu)
I20260620 19:10:13.742421 12958 webserver.cc:460] Webserver started at http://127.9.155.65:38607/ using document root <none> and password file <none>
I20260620 19:10:13.742946 12958 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:13.743008 12958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:13.743199 12958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:10:13.745221 12958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-0/data/instance:
uuid: "f596386dbcc64e95849c193031ce89fe"
format_stamp: "Formatted at 2026-06-20 19:10:13 on dist-test-slave-n3fx"
server_key: "937a09223eb9aaf1651bfc02e8bd0126"
server_key_iv: "2297f156f7b963ba6a8cee78a38d82a3"
server_key_version: "encryptionkey@0"
I20260620 19:10:13.745738 12958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-0/wal/instance:
uuid: "f596386dbcc64e95849c193031ce89fe"
format_stamp: "Formatted at 2026-06-20 19:10:13 on dist-test-slave-n3fx"
server_key: "937a09223eb9aaf1651bfc02e8bd0126"
server_key_iv: "2297f156f7b963ba6a8cee78a38d82a3"
server_key_version: "encryptionkey@0"
I20260620 19:10:13.749286 12958 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 19:10:13.751703 12973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:13.752904 12958 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:10:13.753016 12958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-0/wal
uuid: "f596386dbcc64e95849c193031ce89fe"
format_stamp: "Formatted at 2026-06-20 19:10:13 on dist-test-slave-n3fx"
server_key: "937a09223eb9aaf1651bfc02e8bd0126"
server_key_iv: "2297f156f7b963ba6a8cee78a38d82a3"
server_key_version: "encryptionkey@0"
I20260620 19:10:13.753109 12958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:13.764845 12958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:13.767813 12958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:13.767984 12958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:13.768599 12958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:10:13.769467 12958 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:10:13.769516 12958 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:13.769568 12958 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:10:13.769594 12958 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:13.778056 12958 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.65:40131
I20260620 19:10:13.778083 13086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.65:40131 every 8 connection(s)
I20260620 19:10:13.778898 12958 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
I20260620 19:10:13.781055 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 12958
I20260620 19:10:13.781147 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-0/wal/instance
I20260620 19:10:13.781404 9837 external_mini_cluster.cc:1428] Setting key b9502308149380db4f31d628c2972b0c
Jun 20 19:10:13 dist-test-slave-n3fx krb5kdc[12851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982613, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.65@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.155.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.155.66@KRBTEST.COM" created.
I20260620 19:10:13.792349 12935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:13.780645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.65:39535 (local address 127.9.155.126:43851)
0620 19:10:13.780805 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:13.780810 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:13.781697 (+ 887us) server_negotiation.cc:407] Connection header received
0620 19:10:13.783143 (+ 1446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:13.783146 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:13.783197 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:13.783267 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:13.784476 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:13.785011 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:13.786163 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:13.786413 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:13.786498 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:13.788959 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:13.788963 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:13.788965 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:13.788987 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:13.790284 (+ 1297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:13.790742 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:13.790745 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:13.790747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:13.790859 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:13.791266 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:13.791270 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:13.791272 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:13.791453 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:13.791539 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:13.792019 (+ 480us) server_negotiation.cc:299] Negotiation successful
0620 19:10:13.792155 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 19:10:13.793187 13089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:13.781060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:43851 (local address 127.9.155.65:39535)
0620 19:10:13.781536 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:13.781586 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:13.782948 (+ 1362us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:13.783435 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:13.783442 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:13.783797 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:13.784352 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:13.784367 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:13.785125 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:13.785128 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:13.786049 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:13.786059 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:13.786547 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:13.786551 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:13.786708 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:13.787235 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:13.787251 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:13.788801 (+ 1550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:13.790394 (+ 1593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:13.790400 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:13.790415 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:13.790652 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:13.790984 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:13.790987 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:13.790989 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:13.791174 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:13.791550 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:13.791557 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:13.791795 (+ 238us) client_negotiation.cc:769] Sending connection context
0620 19:10:13.791959 (+ 164us) client_negotiation.cc:240] Negotiation successful
0620 19:10:13.792369 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":118,"threads_started":1}
I20260620 19:10:13.794490 13087 heartbeater.cc:347] Connected to a master server at 127.9.155.126:43851
I20260620 19:10:13.794708 13087 heartbeater.cc:464] Registering TS with master...
I20260620 19:10:13.795176 13087 heartbeater.cc:511] Master 127.9.155.126:43851 requested a full tablet report, sending...
I20260620 19:10:13.796844 12896 ts_manager.cc:194] Registered new tserver with Master: f596386dbcc64e95849c193031ce89fe (127.9.155.65:40131)
I20260620 19:10:13.798393 12896 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.155.65@KRBTEST.COM'} at 127.9.155.65:39535
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.155.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.66.keytab.
Entry for principal kudu/127.9.155.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.66.keytab.
I20260620 19:10:13.807356 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.66:0
--local_ip_for_outbound_sockets=127.9.155.66
--webserver_interface=127.9.155.66
--webserver_port=0
--tserver_master_addrs=127.9.155.126:43851
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.66.keytab
--principal=kudu/127.9.155.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:41489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:10:13.916646 13092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:13.916944 13092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:13.916994 13092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:13.921056 13092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:13.921124 13092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:13.921186 13092 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.66
I20260620 19:10:13.926689 13092 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:41489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.66.keytab
--principal=kudu/127.9.155.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.155.66
--webserver_port=0
--tserver_master_addrs=127.9.155.126:43851
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13092
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.66
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:10:13.927729 13092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:13.928691 13092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:10:13.938032 13098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:13.938040 13097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:13.938053 13100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:13.938189 13092 server_base.cc:1034] running on GCE node
I20260620 19:10:13.939025 13092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:13.939705 13092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:13.940860 13092 hybrid_clock.cc:648] HybridClock initialized: now 1781982613940831 us; error 50 us; skew 500 ppm
Jun 20 19:10:13 dist-test-slave-n3fx krb5kdc[12851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982613, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:13.947044 13092 init.cc:377] Logged in from keytab as kudu/127.9.155.66@KRBTEST.COM (short username kudu)
I20260620 19:10:13.948036 13092 webserver.cc:460] Webserver started at http://127.9.155.66:38489/ using document root <none> and password file <none>
I20260620 19:10:13.948625 13092 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:13.948688 13092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:13.948889 13092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:10:13.950929 13092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-1/data/instance:
uuid: "e60af2d72fed475fa1a000321729fe11"
format_stamp: "Formatted at 2026-06-20 19:10:13 on dist-test-slave-n3fx"
server_key: "28bd10640e87bb7c6a0c28de457e3a04"
server_key_iv: "26ce780ad442464e23d6617fcdf43a25"
server_key_version: "encryptionkey@0"
I20260620 19:10:13.951468 13092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-1/wal/instance:
uuid: "e60af2d72fed475fa1a000321729fe11"
format_stamp: "Formatted at 2026-06-20 19:10:13 on dist-test-slave-n3fx"
server_key: "28bd10640e87bb7c6a0c28de457e3a04"
server_key_iv: "26ce780ad442464e23d6617fcdf43a25"
server_key_version: "encryptionkey@0"
I20260620 19:10:13.955216 13092 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260620 19:10:13.957969 13107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:13.959338 13092 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 19:10:13.959481 13092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-1/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-1/wal
uuid: "e60af2d72fed475fa1a000321729fe11"
format_stamp: "Formatted at 2026-06-20 19:10:13 on dist-test-slave-n3fx"
server_key: "28bd10640e87bb7c6a0c28de457e3a04"
server_key_iv: "26ce780ad442464e23d6617fcdf43a25"
server_key_version: "encryptionkey@0"
I20260620 19:10:13.959606 13092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:13.977422 13092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:13.980516 13092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:13.980683 13092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:13.981251 13092 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:10:13.982059 13092 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:10:13.982103 13092 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:13.982146 13092 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:10:13.982173 13092 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:13.990374 13092 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.66:33919
I20260620 19:10:13.990468 13220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.66:33919 every 8 connection(s)
I20260620 19:10:13.991287 13092 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
I20260620 19:10:13.993364 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 13092
I20260620 19:10:13.993482 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-1/wal/instance
I20260620 19:10:13.993780 9837 external_mini_cluster.cc:1428] Setting key 02973a4e24ad9156402602f46f54102e
Jun 20 19:10:13 dist-test-slave-n3fx krb5kdc[12851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982613, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.66@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.155.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.155.67@KRBTEST.COM" created.
I20260620 19:10:14.005478 12935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:13.993155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.66:49243 (local address 127.9.155.126:43851)
0620 19:10:13.993275 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:13.993278 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:13.994347 (+ 1069us) server_negotiation.cc:407] Connection header received
0620 19:10:13.995787 (+ 1440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:13.995790 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:13.995843 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:13.995930 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:13.997379 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:13.997895 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:13.999066 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:13.999289 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:13.999378 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:14.002062 (+ 2684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:14.002065 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:14.002067 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:14.002091 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:14.003473 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:14.003952 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:14.003955 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:14.003957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:14.004029 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:14.004389 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:14.004393 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:14.004394 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:14.004571 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:14.004667 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:14.005175 (+ 508us) server_negotiation.cc:299] Negotiation successful
0620 19:10:14.005299 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 19:10:14.006314 13223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:13.993624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:43851 (local address 127.9.155.66:49243)
0620 19:10:13.994167 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:13.994228 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:13.995580 (+ 1352us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:13.996089 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:13.996096 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:13.996542 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:13.997213 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:13.997232 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:13.998011 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:13.998013 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:13.998944 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:13.998953 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:13.999439 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:13.999443 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:13.999609 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:14.000233 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:14.000256 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:14.001894 (+ 1638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:14.003583 (+ 1689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:14.003589 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:14.003606 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:14.003852 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:14.004111 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:14.004114 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:14.004116 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:14.004297 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:14.004667 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:14.004674 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:14.004939 (+ 265us) client_negotiation.cc:769] Sending connection context
0620 19:10:14.005101 (+ 162us) client_negotiation.cc:240] Negotiation successful
0620 19:10:14.005433 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":168,"threads_started":1}
I20260620 19:10:14.007668 13221 heartbeater.cc:347] Connected to a master server at 127.9.155.126:43851
I20260620 19:10:14.007918 13221 heartbeater.cc:464] Registering TS with master...
I20260620 19:10:14.008459 13221 heartbeater.cc:511] Master 127.9.155.126:43851 requested a full tablet report, sending...
I20260620 19:10:14.009498 12896 ts_manager.cc:194] Registered new tserver with Master: e60af2d72fed475fa1a000321729fe11 (127.9.155.66:33919)
I20260620 19:10:14.010087 12896 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.155.66@KRBTEST.COM'} at 127.9.155.66:49243
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.155.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.67.keytab.
Entry for principal kudu/127.9.155.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.67.keytab.
I20260620 19:10:14.020771 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.67:0
--local_ip_for_outbound_sockets=127.9.155.67
--webserver_interface=127.9.155.67
--webserver_port=0
--tserver_master_addrs=127.9.155.126:43851
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.67.keytab
--principal=kudu/127.9.155.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:41489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:10:14.129832 13226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:14.130102 13226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:14.130155 13226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:14.134533 13226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:14.134601 13226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:14.134673 13226 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.67
I20260620 19:10:14.140112 13226 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:41489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.67.keytab
--principal=kudu/127.9.155.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.155.67
--webserver_port=0
--tserver_master_addrs=127.9.155.126:43851
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13226
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.67
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:10:14.141243 13226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:14.142256 13226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:10:14.151996 13234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:14.152012 13232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:14.152009 13231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:14.152277 13226 server_base.cc:1034] running on GCE node
I20260620 19:10:14.152762 13226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:14.153400 13226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:14.154563 13226 hybrid_clock.cc:648] HybridClock initialized: now 1781982614154544 us; error 33 us; skew 500 ppm
Jun 20 19:10:14 dist-test-slave-n3fx krb5kdc[12851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982614, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:14.157379 13226 init.cc:377] Logged in from keytab as kudu/127.9.155.67@KRBTEST.COM (short username kudu)
I20260620 19:10:14.158208 13226 webserver.cc:460] Webserver started at http://127.9.155.67:45129/ using document root <none> and password file <none>
I20260620 19:10:14.158725 13226 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:14.158782 13226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:14.158962 13226 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:10:14.160969 13226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-2/data/instance:
uuid: "d87aae8c5ce34af2b17ab40a121eb731"
format_stamp: "Formatted at 2026-06-20 19:10:14 on dist-test-slave-n3fx"
server_key: "3f684921541062e48bc870af41adb319"
server_key_iv: "6c5f4ffab3116bbfdca36fc3819f9669"
server_key_version: "encryptionkey@0"
I20260620 19:10:14.161466 13226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-2/wal/instance:
uuid: "d87aae8c5ce34af2b17ab40a121eb731"
format_stamp: "Formatted at 2026-06-20 19:10:14 on dist-test-slave-n3fx"
server_key: "3f684921541062e48bc870af41adb319"
server_key_iv: "6c5f4ffab3116bbfdca36fc3819f9669"
server_key_version: "encryptionkey@0"
I20260620 19:10:14.165352 13226 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 19:10:14.167882 13241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:14.169266 13226 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:10:14.169394 13226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-2/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-2/wal
uuid: "d87aae8c5ce34af2b17ab40a121eb731"
format_stamp: "Formatted at 2026-06-20 19:10:14 on dist-test-slave-n3fx"
server_key: "3f684921541062e48bc870af41adb319"
server_key_iv: "6c5f4ffab3116bbfdca36fc3819f9669"
server_key_version: "encryptionkey@0"
I20260620 19:10:14.169517 13226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:14.191157 13226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:14.194584 13226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:14.194761 13226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:14.195415 13226 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:10:14.196274 13226 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:10:14.196321 13226 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:14.196365 13226 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:10:14.196394 13226 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:14.205188 13226 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.67:34351
I20260620 19:10:14.205214 13354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.67:34351 every 8 connection(s)
I20260620 19:10:14.206049 13226 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
I20260620 19:10:14.206866 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 13226
I20260620 19:10:14.206990 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982595231230-9837-0/minicluster-data/ts-2/wal/instance
I20260620 19:10:14.207297 9837 external_mini_cluster.cc:1428] Setting key 1542630b7e3a48cea1e25a856b879933
Jun 20 19:10:14 dist-test-slave-n3fx krb5kdc[12851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982614, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.67@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:10:14.221441 12935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:14.208262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.67:42111 (local address 127.9.155.126:43851)
0620 19:10:14.208465 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:14.208469 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:14.209333 (+ 864us) server_negotiation.cc:407] Connection header received
0620 19:10:14.210691 (+ 1358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:14.210694 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:14.210739 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:14.210811 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:14.212509 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:14.213053 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:14.214807 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:14.215040 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:14.215130 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:14.217767 (+ 2637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:14.217770 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:14.217772 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:14.217802 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:14.219173 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:14.219788 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:14.219791 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:14.219792 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:14.219876 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:14.220207 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:14.220210 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:14.220235 (+ 25us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:14.220456 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:14.220541 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:14.221141 (+ 600us) server_negotiation.cc:299] Negotiation successful
0620 19:10:14.221277 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260620 19:10:14.222378 13357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:14.208655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:43851 (local address 127.9.155.67:42111)
0620 19:10:14.209169 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:14.209231 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:14.210463 (+ 1232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:14.211015 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:14.211025 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:14.211570 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:14.212341 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:14.212366 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:14.213190 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:14.213194 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:14.214657 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:14.214670 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:14.215208 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:14.215213 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:14.215434 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:14.216059 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:14.216076 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:14.217581 (+ 1505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:14.219328 (+ 1747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:14.219333 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:14.219389 (+ 56us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:14.219662 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:14.219946 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:14.219948 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:14.219950 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:14.220081 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:14.220579 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:14.220584 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:14.220861 (+ 277us) client_negotiation.cc:769] Sending connection context
0620 19:10:14.221042 (+ 181us) client_negotiation.cc:240] Negotiation successful
0620 19:10:14.221407 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":123,"threads_started":1}
I20260620 19:10:14.223497 13355 heartbeater.cc:347] Connected to a master server at 127.9.155.126:43851
I20260620 19:10:14.223747 13355 heartbeater.cc:464] Registering TS with master...
I20260620 19:10:14.224386 13355 heartbeater.cc:511] Master 127.9.155.126:43851 requested a full tablet report, sending...
I20260620 19:10:14.225446 12896 ts_manager.cc:194] Registered new tserver with Master: d87aae8c5ce34af2b17ab40a121eb731 (127.9.155.67:34351)
I20260620 19:10:14.225992 12896 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.155.67@KRBTEST.COM'} at 127.9.155.67:42111
I20260620 19:10:14.234570 9837 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 19:10:14 dist-test-slave-n3fx krb5kdc[12851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982614, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 20 19:10:14 dist-test-slave-n3fx krb5kdc[12851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982614, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:10:14.259738 12935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:14.251176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56444 (local address 127.9.155.126:43851)
0620 19:10:14.251332 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:14.251341 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:14.251504 (+ 163us) server_negotiation.cc:407] Connection header received
0620 19:10:14.251598 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:14.251601 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:14.251648 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:14.251722 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:14.252413 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:14.252954 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:14.253827 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:14.254054 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:14.254136 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:14.256580 (+ 2444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:14.256584 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:14.256586 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:14.256613 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:14.257957 (+ 1344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:14.258462 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:14.258466 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:14.258468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:14.258582 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:14.258838 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:14.258841 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:14.258843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:14.259082 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:14.259205 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:14.259445 (+ 240us) server_negotiation.cc:299] Negotiation successful
0620 19:10:14.259583 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260620 19:10:14.260134 12896 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:56444
I20260620 19:10:14.260977 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 12958
I20260620 19:10:14.266558 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 13092
I20260620 19:10:14.271517 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 13226
I20260620 19:10:14.276015 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 12864
2026-06-20T19:10:14Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2047 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:10:14 dist-test-slave-n3fx krb5kdc[13367](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:10:14 dist-test-slave-n3fx krb5kdc[13367](info): set up 2 sockets
Jun 20 19:10:14 dist-test-slave-n3fx krb5kdc[13367](info): commencing operation
krb5kdc: starting...
W20260620 19:10:15.232280 9837 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/krb5kdc/test-user.keytab.
Jun 20 19:10:15 dist-test-slave-n3fx krb5kdc[13367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982615, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:10:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:10:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.155.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.155.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.155.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab.
Entry for principal kudu/127.9.155.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab.
I20260620 19:10:15.364507 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:39121
--webserver_interface=127.9.155.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab
--principal=kudu/127.9.155.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:42795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.155.126:39121
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:10:15.479396 13380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:15.479700 13380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:15.479759 13380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:15.484107 13380 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:10:15.484237 13380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:15.484269 13380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:15.484305 13380 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:10:15.484334 13380 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:10:15.490061 13380 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:42795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.155.126:39121
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab
--principal=kudu/127.9.155.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:39121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.155.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13380
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:10:15.491348 13380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:15.492519 13380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:10:15.502099 13386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:15.502102 13385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:15.502102 13388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:15.502146 13380 server_base.cc:1034] running on GCE node
I20260620 19:10:15.503051 13380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:15.504282 13380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:15.505496 13380 hybrid_clock.cc:648] HybridClock initialized: now 1781982615505447 us; error 67 us; skew 500 ppm
Jun 20 19:10:15 dist-test-slave-n3fx krb5kdc[13367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982615, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:15.508674 13380 init.cc:377] Logged in from keytab as kudu/127.9.155.126@KRBTEST.COM (short username kudu)
I20260620 19:10:15.509717 13380 webserver.cc:460] Webserver started at http://127.9.155.126:35257/ using document root <none> and password file <none>
I20260620 19:10:15.510277 13380 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:15.510339 13380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:15.510524 13380 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:10:15.512944 13380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/minicluster-data/master-0/data/instance:
uuid: "f10ea7bac51d451ba7fa8ea2cadc43ba"
format_stamp: "Formatted at 2026-06-20 19:10:15 on dist-test-slave-n3fx"
server_key: "16d10b2900aba86eb9c2b07fd2df592c"
server_key_iv: "f53d9fe8e93315eb86d54140d1a168f6"
server_key_version: "encryptionkey@0"
I20260620 19:10:15.513664 13380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/minicluster-data/master-0/wal/instance:
uuid: "f10ea7bac51d451ba7fa8ea2cadc43ba"
format_stamp: "Formatted at 2026-06-20 19:10:15 on dist-test-slave-n3fx"
server_key: "16d10b2900aba86eb9c2b07fd2df592c"
server_key_iv: "f53d9fe8e93315eb86d54140d1a168f6"
server_key_version: "encryptionkey@0"
I20260620 19:10:15.518092 13380 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 19:10:15.520939 13395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:15.522365 13380 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:10:15.522547 13380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/minicluster-data/master-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/minicluster-data/master-0/wal
uuid: "f10ea7bac51d451ba7fa8ea2cadc43ba"
format_stamp: "Formatted at 2026-06-20 19:10:15 on dist-test-slave-n3fx"
server_key: "16d10b2900aba86eb9c2b07fd2df592c"
server_key_iv: "f53d9fe8e93315eb86d54140d1a168f6"
server_key_version: "encryptionkey@0"
I20260620 19:10:15.522684 13380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:15.547511 13380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:15.551065 13380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:15.551262 13380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:15.558729 13380 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.126:39121
I20260620 19:10:15.558804 13447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.126:39121 every 8 connection(s)
I20260620 19:10:15.559854 13380 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
I20260620 19:10:15.561326 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 13380
I20260620 19:10:15.561421 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/minicluster-data/master-0/wal/instance
I20260620 19:10:15.561733 9837 external_mini_cluster.cc:1428] Setting key 3cfb21032a81824493e89a55f8f57306
I20260620 19:10:15.563140 13448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:15.569432 13448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f10ea7bac51d451ba7fa8ea2cadc43ba: Bootstrap starting.
Jun 20 19:10:15 dist-test-slave-n3fx krb5kdc[13367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982615, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:10:15.572201 13448 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f10ea7bac51d451ba7fa8ea2cadc43ba: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:15.573148 13448 log.cc:824] T 00000000000000000000000000000000 P f10ea7bac51d451ba7fa8ea2cadc43ba: Log is configured to *not* fsync() on all Append() calls
I20260620 19:10:15.575193 13448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f10ea7bac51d451ba7fa8ea2cadc43ba: No bootstrap required, opened a new log
I20260620 19:10:15.577728 13451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:15.563291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57436 (local address 127.9.155.126:39121)
0620 19:10:15.563795 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:15.563963 (+ 168us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:15.564004 (+ 41us) server_negotiation.cc:407] Connection header received
0620 19:10:15.564897 (+ 893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:15.564919 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:15.565389 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:15.565732 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:15.566568 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:15.567583 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:15.568524 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:15.569032 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:15.569182 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:15.572077 (+ 2895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:15.572082 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:15.572098 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:15.572133 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:15.573997 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:15.574461 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:15.574466 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:15.574472 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:15.574585 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:15.574875 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:15.574877 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:15.574878 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:15.575287 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:15.575476 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:15.575952 (+ 476us) server_negotiation.cc:299] Negotiation successful
0620 19:10:15.576310 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":158,"threads_started":1}
I20260620 19:10:15.579341 13448 raft_consensus.cc:348] T 00000000000000000000000000000000 P f10ea7bac51d451ba7fa8ea2cadc43ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f10ea7bac51d451ba7fa8ea2cadc43ba" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 39121 } }
I20260620 19:10:15.579603 13448 raft_consensus.cc:374] T 00000000000000000000000000000000 P f10ea7bac51d451ba7fa8ea2cadc43ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:15.579666 13448 raft_consensus.cc:729] T 00000000000000000000000000000000 P f10ea7bac51d451ba7fa8ea2cadc43ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f10ea7bac51d451ba7fa8ea2cadc43ba, State: Initialized, Role: FOLLOWER
I20260620 19:10:15.580168 13448 consensus_queue.cc:260] T 00000000000000000000000000000000 P f10ea7bac51d451ba7fa8ea2cadc43ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f10ea7bac51d451ba7fa8ea2cadc43ba" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 39121 } }
I20260620 19:10:15.580367 13448 raft_consensus.cc:388] T 00000000000000000000000000000000 P f10ea7bac51d451ba7fa8ea2cadc43ba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:10:15.580446 13448 raft_consensus.cc:482] T 00000000000000000000000000000000 P f10ea7bac51d451ba7fa8ea2cadc43ba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:10:15.580546 13448 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f10ea7bac51d451ba7fa8ea2cadc43ba [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:15.581534 13448 raft_consensus.cc:504] T 00000000000000000000000000000000 P f10ea7bac51d451ba7fa8ea2cadc43ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f10ea7bac51d451ba7fa8ea2cadc43ba" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 39121 } }
I20260620 19:10:15.581887 13448 leader_election.cc:302] T 00000000000000000000000000000000 P f10ea7bac51d451ba7fa8ea2cadc43ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f10ea7bac51d451ba7fa8ea2cadc43ba; no voters:
I20260620 19:10:15.582212 13448 leader_election.cc:288] T 00000000000000000000000000000000 P f10ea7bac51d451ba7fa8ea2cadc43ba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:10:15.582381 13453 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f10ea7bac51d451ba7fa8ea2cadc43ba [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:10:15.582670 13453 raft_consensus.cc:686] T 00000000000000000000000000000000 P f10ea7bac51d451ba7fa8ea2cadc43ba [term 1 LEADER]: Becoming Leader. State: Replica: f10ea7bac51d451ba7fa8ea2cadc43ba, State: Running, Role: LEADER
I20260620 19:10:15.583011 13453 consensus_queue.cc:237] T 00000000000000000000000000000000 P f10ea7bac51d451ba7fa8ea2cadc43ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f10ea7bac51d451ba7fa8ea2cadc43ba" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 39121 } }
I20260620 19:10:15.583493 13448 sys_catalog.cc:564] T 00000000000000000000000000000000 P f10ea7bac51d451ba7fa8ea2cadc43ba [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:10:15.584748 13454 sys_catalog.cc:455] T 00000000000000000000000000000000 P f10ea7bac51d451ba7fa8ea2cadc43ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f10ea7bac51d451ba7fa8ea2cadc43ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f10ea7bac51d451ba7fa8ea2cadc43ba" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 39121 } } }
I20260620 19:10:15.584910 13454 sys_catalog.cc:458] T 00000000000000000000000000000000 P f10ea7bac51d451ba7fa8ea2cadc43ba [sys.catalog]: This master's current role is: LEADER
I20260620 19:10:15.585269 13462 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:10:15.585471 13455 sys_catalog.cc:455] T 00000000000000000000000000000000 P f10ea7bac51d451ba7fa8ea2cadc43ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader f10ea7bac51d451ba7fa8ea2cadc43ba. Latest consensus state: current_term: 1 leader_uuid: "f10ea7bac51d451ba7fa8ea2cadc43ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f10ea7bac51d451ba7fa8ea2cadc43ba" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 39121 } } }
I20260620 19:10:15.585662 13455 sys_catalog.cc:458] T 00000000000000000000000000000000 P f10ea7bac51d451ba7fa8ea2cadc43ba [sys.catalog]: This master's current role is: LEADER
I20260620 19:10:15.588711 13462 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:10:15.594645 13462 catalog_manager.cc:1347] Generated new cluster ID: 431eeafe98be44bb8a718d5a37ed27f1
I20260620 19:10:15.594726 13462 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:10:15.606153 13462 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:10:15.607208 13462 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:10:15.613715 13462 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f10ea7bac51d451ba7fa8ea2cadc43ba: Generated new TSK 0
I20260620 19:10:15.614391 13462 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.155.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.155.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.155.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.65.keytab.
Entry for principal kudu/127.9.155.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.65.keytab.
I20260620 19:10:15.648429 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.65:0
--local_ip_for_outbound_sockets=127.9.155.65
--webserver_interface=127.9.155.65
--webserver_port=0
--tserver_master_addrs=127.9.155.126:39121
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.65.keytab
--principal=kudu/127.9.155.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:42795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:10:15.762337 13474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:15.762615 13474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:15.762660 13474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:15.766965 13474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:15.767042 13474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:15.767104 13474 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.65
I20260620 19:10:15.772601 13474 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:42795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.65.keytab
--principal=kudu/127.9.155.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.155.65
--webserver_port=0
--tserver_master_addrs=127.9.155.126:39121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.65
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:10:15.773696 13474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:15.774660 13474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:10:15.784072 13480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:15.784292 13474 server_base.cc:1034] running on GCE node
W20260620 19:10:15.784091 13479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:15.784066 13482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:15.784801 13474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:15.785425 13474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:15.786593 13474 hybrid_clock.cc:648] HybridClock initialized: now 1781982615786573 us; error 31 us; skew 500 ppm
Jun 20 19:10:15 dist-test-slave-n3fx krb5kdc[13367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982615, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:15.789435 13474 init.cc:377] Logged in from keytab as kudu/127.9.155.65@KRBTEST.COM (short username kudu)
I20260620 19:10:15.790329 13474 webserver.cc:460] Webserver started at http://127.9.155.65:35863/ using document root <none> and password file <none>
I20260620 19:10:15.790853 13474 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:15.790915 13474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:15.791091 13474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:10:15.793159 13474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/minicluster-data/ts-0/data/instance:
uuid: "b75d1cb5a71c493e9232f205a069aee8"
format_stamp: "Formatted at 2026-06-20 19:10:15 on dist-test-slave-n3fx"
server_key: "5ef14aef1714890f31e88922ff2cf3fe"
server_key_iv: "532294cc7b4fb240abb1255ecb7c3e92"
server_key_version: "encryptionkey@0"
I20260620 19:10:15.793671 13474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/minicluster-data/ts-0/wal/instance:
uuid: "b75d1cb5a71c493e9232f205a069aee8"
format_stamp: "Formatted at 2026-06-20 19:10:15 on dist-test-slave-n3fx"
server_key: "5ef14aef1714890f31e88922ff2cf3fe"
server_key_iv: "532294cc7b4fb240abb1255ecb7c3e92"
server_key_version: "encryptionkey@0"
I20260620 19:10:15.797480 13474 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260620 19:10:15.799990 13489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:15.801453 13474 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 19:10:15.801561 13474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/minicluster-data/ts-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/minicluster-data/ts-0/wal
uuid: "b75d1cb5a71c493e9232f205a069aee8"
format_stamp: "Formatted at 2026-06-20 19:10:15 on dist-test-slave-n3fx"
server_key: "5ef14aef1714890f31e88922ff2cf3fe"
server_key_iv: "532294cc7b4fb240abb1255ecb7c3e92"
server_key_version: "encryptionkey@0"
I20260620 19:10:15.801673 13474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:15.822422 13474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:15.825533 13474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:15.825704 13474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:15.826303 13474 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:10:15.827129 13474 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:10:15.827178 13474 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:15.827232 13474 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:10:15.827256 13474 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:15.835273 13474 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.65:34395
I20260620 19:10:15.835326 13602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.65:34395 every 8 connection(s)
I20260620 19:10:15.836145 13474 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
I20260620 19:10:15.844755 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 13474
I20260620 19:10:15.844866 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/minicluster-data/ts-0/wal/instance
I20260620 19:10:15.845155 9837 external_mini_cluster.cc:1428] Setting key 74db60c53d3ea3251bc2a308d506d9d4
Jun 20 19:10:15 dist-test-slave-n3fx krb5kdc[13367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982615, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.65@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:10:15.849404 13451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:15.837926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.65:57407 (local address 127.9.155.126:39121)
0620 19:10:15.838056 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:15.838059 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:15.838832 (+ 773us) server_negotiation.cc:407] Connection header received
0620 19:10:15.840133 (+ 1301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:15.840136 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:15.840183 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:15.840286 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:15.841444 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:15.842003 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:15.843169 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:15.843406 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:15.843493 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:15.846021 (+ 2528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:15.846025 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:15.846027 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:15.846051 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:15.847363 (+ 1312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:15.847908 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:15.847913 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:15.847915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:15.847999 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:15.848357 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:15.848360 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:15.848361 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:15.848527 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:15.848600 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:15.849176 (+ 576us) server_negotiation.cc:299] Negotiation successful
0620 19:10:15.849280 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 19:10:15.850531 13605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:15.838261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:39121 (local address 127.9.155.65:57407)
0620 19:10:15.838687 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:15.838727 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:15.839943 (+ 1216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:15.840434 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:15.840441 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:15.840786 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:15.841283 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:15.841296 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:15.842122 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:15.842125 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:15.843041 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:15.843052 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:15.843541 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:15.843544 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:15.843692 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:15.844298 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:15.844323 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:15.845847 (+ 1524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:15.847473 (+ 1626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:15.847478 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:15.847495 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:15.847753 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:15.848079 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:15.848082 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:15.848084 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:15.848281 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:15.848614 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:15.848620 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:15.848947 (+ 327us) client_negotiation.cc:769] Sending connection context
0620 19:10:15.849142 (+ 195us) client_negotiation.cc:240] Negotiation successful
0620 19:10:15.849481 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":119,"threads_started":1}
I20260620 19:10:15.851758 13603 heartbeater.cc:347] Connected to a master server at 127.9.155.126:39121
I20260620 19:10:15.851991 13603 heartbeater.cc:464] Registering TS with master...
I20260620 19:10:15.852488 13603 heartbeater.cc:511] Master 127.9.155.126:39121 requested a full tablet report, sending...
I20260620 19:10:15.853928 13410 ts_manager.cc:194] Registered new tserver with Master: b75d1cb5a71c493e9232f205a069aee8 (127.9.155.65:34395)
WARNING: no policy specified for kudu/127.9.155.66@KRBTEST.COM; defaulting to no policy
I20260620 19:10:15.855008 13410 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.155.65@KRBTEST.COM'} at 127.9.155.65:57407
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.155.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.155.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.66.keytab.
Entry for principal kudu/127.9.155.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.66.keytab.
I20260620 19:10:15.871304 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.66:0
--local_ip_for_outbound_sockets=127.9.155.66
--webserver_interface=127.9.155.66
--webserver_port=0
--tserver_master_addrs=127.9.155.126:39121
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.66.keytab
--principal=kudu/127.9.155.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:42795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:10:15.981537 13608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:15.981788 13608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:15.981848 13608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:15.986056 13608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:15.986123 13608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:15.986202 13608 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.66
I20260620 19:10:15.991662 13608 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:42795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.66.keytab
--principal=kudu/127.9.155.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.155.66
--webserver_port=0
--tserver_master_addrs=127.9.155.126:39121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13608
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.66
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:10:15.992828 13608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:15.993850 13608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:10:16.003509 13614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:16.003548 13608 server_base.cc:1034] running on GCE node
W20260620 19:10:16.003509 13616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:16.003516 13613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:16.004155 13608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:16.004846 13608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:16.006004 13608 hybrid_clock.cc:648] HybridClock initialized: now 1781982616005984 us; error 32 us; skew 500 ppm
Jun 20 19:10:16 dist-test-slave-n3fx krb5kdc[13367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982616, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:16.008724 13608 init.cc:377] Logged in from keytab as kudu/127.9.155.66@KRBTEST.COM (short username kudu)
I20260620 19:10:16.009575 13608 webserver.cc:460] Webserver started at http://127.9.155.66:43061/ using document root <none> and password file <none>
I20260620 19:10:16.010097 13608 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:16.010159 13608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:16.010344 13608 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:10:16.012408 13608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/minicluster-data/ts-1/data/instance:
uuid: "b2622570f31e454f96edbfbed40aafa7"
format_stamp: "Formatted at 2026-06-20 19:10:16 on dist-test-slave-n3fx"
server_key: "e9e114b0bfe04e63d81966f8bec134e4"
server_key_iv: "93982a8c666e75c2e3d8eeba2b161c57"
server_key_version: "encryptionkey@0"
I20260620 19:10:16.012938 13608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/minicluster-data/ts-1/wal/instance:
uuid: "b2622570f31e454f96edbfbed40aafa7"
format_stamp: "Formatted at 2026-06-20 19:10:16 on dist-test-slave-n3fx"
server_key: "e9e114b0bfe04e63d81966f8bec134e4"
server_key_iv: "93982a8c666e75c2e3d8eeba2b161c57"
server_key_version: "encryptionkey@0"
I20260620 19:10:16.016606 13608 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 19:10:16.019047 13623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:16.020342 13608 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:10:16.020499 13608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/minicluster-data/ts-1/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/minicluster-data/ts-1/wal
uuid: "b2622570f31e454f96edbfbed40aafa7"
format_stamp: "Formatted at 2026-06-20 19:10:16 on dist-test-slave-n3fx"
server_key: "e9e114b0bfe04e63d81966f8bec134e4"
server_key_iv: "93982a8c666e75c2e3d8eeba2b161c57"
server_key_version: "encryptionkey@0"
I20260620 19:10:16.020604 13608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:16.038977 13608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:16.042161 13608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:16.042358 13608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:16.043057 13608 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:10:16.043972 13608 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:10:16.044023 13608 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:16.044078 13608 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:10:16.044108 13608 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:16.052876 13608 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.66:36765
I20260620 19:10:16.052917 13736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.66:36765 every 8 connection(s)
I20260620 19:10:16.053845 13608 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
I20260620 19:10:16.057660 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 13608
I20260620 19:10:16.057762 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/minicluster-data/ts-1/wal/instance
I20260620 19:10:16.058012 9837 external_mini_cluster.cc:1428] Setting key c3cb3e9a95ca6449f2334cd294eb1ece
Jun 20 19:10:16 dist-test-slave-n3fx krb5kdc[13367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982616, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.66@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.155.67@KRBTEST.COM; defaulting to no policy
I20260620 19:10:16.068357 13451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:16.055834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.66:53605 (local address 127.9.155.126:39121)
0620 19:10:16.055996 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:16.056000 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:16.056954 (+ 954us) server_negotiation.cc:407] Connection header received
0620 19:10:16.058437 (+ 1483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:16.058440 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:16.058500 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:16.058577 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:16.059940 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:16.060507 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:16.061711 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:16.061938 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:16.062029 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:16.064594 (+ 2565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:16.064598 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:16.064600 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:16.064626 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:16.066023 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:16.066598 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:16.066601 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:16.066602 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:16.066680 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:16.067002 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:16.067005 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:16.067007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:16.067201 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:16.067322 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:16.067969 (+ 647us) server_negotiation.cc:299] Negotiation successful
0620 19:10:16.068152 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.155.67@KRBTEST.COM" created.
I20260620 19:10:16.069367 13739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:16.056268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:39121 (local address 127.9.155.66:53605)
0620 19:10:16.056759 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:16.056802 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:16.058210 (+ 1408us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:16.058770 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:16.058778 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:16.059196 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:16.059771 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:16.059786 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:16.060641 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:16.060644 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:16.061577 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:16.061587 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:16.062092 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:16.062096 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:16.062275 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:16.062901 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:16.062925 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:16.064411 (+ 1486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:16.066128 (+ 1717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:16.066134 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:16.066153 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:16.066494 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:16.066762 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:16.066765 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:16.066767 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:16.066914 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:16.067359 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:16.067367 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:16.067666 (+ 299us) client_negotiation.cc:769] Sending connection context
0620 19:10:16.067892 (+ 226us) client_negotiation.cc:240] Negotiation successful
0620 19:10:16.068357 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":127,"threads_started":1}
I20260620 19:10:16.070734 13737 heartbeater.cc:347] Connected to a master server at 127.9.155.126:39121
I20260620 19:10:16.071061 13737 heartbeater.cc:464] Registering TS with master...
I20260620 19:10:16.071820 13737 heartbeater.cc:511] Master 127.9.155.126:39121 requested a full tablet report, sending...
I20260620 19:10:16.073065 13410 ts_manager.cc:194] Registered new tserver with Master: b2622570f31e454f96edbfbed40aafa7 (127.9.155.66:36765)
I20260620 19:10:16.073683 13410 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.155.66@KRBTEST.COM'} at 127.9.155.66:53605
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.155.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.67.keytab.
Entry for principal kudu/127.9.155.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.67.keytab.
I20260620 19:10:16.086999 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.67:0
--local_ip_for_outbound_sockets=127.9.155.67
--webserver_interface=127.9.155.67
--webserver_port=0
--tserver_master_addrs=127.9.155.126:39121
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.67.keytab
--principal=kudu/127.9.155.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:42795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:10:16.198732 13742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:16.199015 13742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:16.199059 13742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:16.203295 13742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:16.203363 13742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:16.203436 13742 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.67
I20260620 19:10:16.208981 13742 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:42795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.67.keytab
--principal=kudu/127.9.155.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.155.67
--webserver_port=0
--tserver_master_addrs=127.9.155.126:39121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13742
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.67
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:10:16.210099 13742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:16.211325 13742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:10:16.221941 13750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:16.221940 13747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:16.221932 13748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:16.222800 13742 server_base.cc:1034] running on GCE node
I20260620 19:10:16.223253 13742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:16.223971 13742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:16.225128 13742 hybrid_clock.cc:648] HybridClock initialized: now 1781982616225085 us; error 58 us; skew 500 ppm
Jun 20 19:10:16 dist-test-slave-n3fx krb5kdc[13367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982616, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:16.228020 13742 init.cc:377] Logged in from keytab as kudu/127.9.155.67@KRBTEST.COM (short username kudu)
I20260620 19:10:16.229024 13742 webserver.cc:460] Webserver started at http://127.9.155.67:43595/ using document root <none> and password file <none>
I20260620 19:10:16.229573 13742 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:16.229641 13742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:16.229838 13742 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:10:16.231906 13742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/minicluster-data/ts-2/data/instance:
uuid: "636f9c34ca1f461b992cc3884d2cc116"
format_stamp: "Formatted at 2026-06-20 19:10:16 on dist-test-slave-n3fx"
server_key: "c83dff821acefd6de33facac1e9c0a0e"
server_key_iv: "c6dc7f98a91638c4743f13a15232a9fe"
server_key_version: "encryptionkey@0"
I20260620 19:10:16.232462 13742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/minicluster-data/ts-2/wal/instance:
uuid: "636f9c34ca1f461b992cc3884d2cc116"
format_stamp: "Formatted at 2026-06-20 19:10:16 on dist-test-slave-n3fx"
server_key: "c83dff821acefd6de33facac1e9c0a0e"
server_key_iv: "c6dc7f98a91638c4743f13a15232a9fe"
server_key_version: "encryptionkey@0"
I20260620 19:10:16.236346 13742 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 19:10:16.238929 13757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:16.240190 13742 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 19:10:16.240363 13742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/minicluster-data/ts-2/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/minicluster-data/ts-2/wal
uuid: "636f9c34ca1f461b992cc3884d2cc116"
format_stamp: "Formatted at 2026-06-20 19:10:16 on dist-test-slave-n3fx"
server_key: "c83dff821acefd6de33facac1e9c0a0e"
server_key_iv: "c6dc7f98a91638c4743f13a15232a9fe"
server_key_version: "encryptionkey@0"
I20260620 19:10:16.240470 13742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:16.260435 13742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:16.263680 13742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:16.263871 13742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:16.264477 13742 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:10:16.265277 13742 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:10:16.265324 13742 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:16.265375 13742 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:10:16.265404 13742 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:16.273792 13742 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.67:39879
I20260620 19:10:16.273869 13870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.67:39879 every 8 connection(s)
I20260620 19:10:16.274749 13742 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
I20260620 19:10:16.283655 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 13742
I20260620 19:10:16.283766 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982595231230-9837-0/minicluster-data/ts-2/wal/instance
I20260620 19:10:16.283983 9837 external_mini_cluster.cc:1428] Setting key e217d5a830e4d747c915868634b62024
Jun 20 19:10:16 dist-test-slave-n3fx krb5kdc[13367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982616, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.67@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:10:16.289515 13451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:16.276638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.67:39205 (local address 127.9.155.126:39121)
0620 19:10:16.276777 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:16.276781 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:16.277645 (+ 864us) server_negotiation.cc:407] Connection header received
0620 19:10:16.279310 (+ 1665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:16.279313 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:16.279374 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:16.279456 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:16.280709 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:16.281242 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:16.282384 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:16.282597 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:16.282678 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:16.286255 (+ 3577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:16.286259 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:16.286262 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:16.286294 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:16.287521 (+ 1227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:16.288013 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:16.288016 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:16.288017 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:16.288088 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:16.288447 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:16.288449 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:16.288450 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:16.288624 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:16.288699 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:16.289273 (+ 574us) server_negotiation.cc:299] Negotiation successful
0620 19:10:16.289389 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 19:10:16.290382 13873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:16.277007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:39121 (local address 127.9.155.67:39205)
0620 19:10:16.277481 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:16.277525 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:16.279067 (+ 1542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:16.279638 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:16.279646 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:16.280020 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:16.280569 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:16.280583 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:16.281365 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:16.281369 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:16.282250 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:16.282260 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:16.282737 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:16.282741 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:16.282902 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:16.284239 (+ 1337us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:16.284263 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:16.286108 (+ 1845us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:16.287626 (+ 1518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:16.287631 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:16.287644 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:16.287922 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:16.288163 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:16.288166 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:16.288167 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:16.288369 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:16.288744 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:16.288750 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:16.289012 (+ 262us) client_negotiation.cc:769] Sending connection context
0620 19:10:16.289180 (+ 168us) client_negotiation.cc:240] Negotiation successful
0620 19:10:16.289511 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":119,"threads_started":1}
I20260620 19:10:16.291626 13871 heartbeater.cc:347] Connected to a master server at 127.9.155.126:39121
I20260620 19:10:16.291893 13871 heartbeater.cc:464] Registering TS with master...
I20260620 19:10:16.292488 13871 heartbeater.cc:511] Master 127.9.155.126:39121 requested a full tablet report, sending...
I20260620 19:10:16.293601 13410 ts_manager.cc:194] Registered new tserver with Master: 636f9c34ca1f461b992cc3884d2cc116 (127.9.155.67:39879)
I20260620 19:10:16.294191 13410 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.155.67@KRBTEST.COM'} at 127.9.155.67:39205
I20260620 19:10:16.298152 9837 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 19:10:16 dist-test-slave-n3fx krb5kdc[13367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 19:10:16 dist-test-slave-n3fx krb5kdc[13367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.155.65@KRBTEST.COM
I20260620 19:10:16.326030 13878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:16.313907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39102 (local address 127.9.155.65:34395)
0620 19:10:16.314230 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:16.314246 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:16.314278 (+ 32us) server_negotiation.cc:407] Connection header received
0620 19:10:16.314329 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:16.314334 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:16.314545 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:16.314701 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:16.315595 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:16.316878 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:16.318153 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:16.318721 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:16.318843 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:16.321752 (+ 2909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:16.321763 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:16.321779 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:16.321809 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:16.323822 (+ 2013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:16.324397 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:16.324403 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:16.324405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:16.324502 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:16.324900 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:16.324904 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:16.324905 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:16.325200 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:16.325391 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:16.325665 (+ 274us) server_negotiation.cc:299] Negotiation successful
0620 19:10:16.325796 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":98,"threads_started":1}
Jun 20 19:10:16 dist-test-slave-n3fx krb5kdc[13367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:10:16.337199 13451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:16.329202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57452 (local address 127.9.155.126:39121)
0620 19:10:16.329341 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:16.329344 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:16.329359 (+ 15us) server_negotiation.cc:407] Connection header received
0620 19:10:16.329517 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:16.329519 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:16.329563 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:16.329627 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:16.330198 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:16.330712 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:16.331522 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:16.331741 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:16.331830 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:16.334221 (+ 2391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:16.334224 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:16.334226 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:16.334252 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:16.335584 (+ 1332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:16.336002 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:16.336005 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:16.336008 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:16.336085 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:16.336365 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:16.336368 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:16.336370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:16.336597 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:16.336718 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:16.336936 (+ 218us) server_negotiation.cc:299] Negotiation successful
0620 19:10:16.337047 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260620 19:10:16.337598 13410 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57452
I20260620 19:10:16.338265 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 13474
I20260620 19:10:16.343116 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 13608
I20260620 19:10:16.348017 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 13742
I20260620 19:10:16.352803 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 13380
2026-06-20T19:10:16Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2076 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:10:16 dist-test-slave-n3fx krb5kdc[13883](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:10:16 dist-test-slave-n3fx krb5kdc[13883](info): set up 2 sockets
Jun 20 19:10:16 dist-test-slave-n3fx krb5kdc[13883](info): commencing operation
krb5kdc: starting...
W20260620 19:10:17.308087 9837 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/krb5kdc/test-user.keytab.
Jun 20 19:10:17 dist-test-slave-n3fx krb5kdc[13883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:10:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:10:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.155.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.155.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.155.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab.
Entry for principal kudu/127.9.155.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab.
I20260620 19:10:17.423013 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:38073
--webserver_interface=127.9.155.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab
--principal=kudu/127.9.155.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:36915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.155.126:38073
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:10:17.534183 13896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:17.534473 13896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:17.534536 13896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:17.538676 13896 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:10:17.538748 13896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:17.538774 13896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:17.538794 13896 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:10:17.538823 13896 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:10:17.544178 13896 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:36915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.155.126:38073
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab
--principal=kudu/127.9.155.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:38073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.155.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13896
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:10:17.545262 13896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:17.546280 13896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:10:17.554320 13904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:17.554389 13901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:17.554555 13896 server_base.cc:1034] running on GCE node
W20260620 19:10:17.554250 13902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:17.555164 13896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:17.556238 13896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:17.557395 13896 hybrid_clock.cc:648] HybridClock initialized: now 1781982617557369 us; error 40 us; skew 500 ppm
Jun 20 19:10:17 dist-test-slave-n3fx krb5kdc[13883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982617, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:17.559988 13896 init.cc:377] Logged in from keytab as kudu/127.9.155.126@KRBTEST.COM (short username kudu)
I20260620 19:10:17.560714 13896 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:17.560782 13896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:17.560971 13896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:10:17.563015 13896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/minicluster-data/master-0/data/instance:
uuid: "815622646e3045c889d0f94cd715cade"
format_stamp: "Formatted at 2026-06-20 19:10:17 on dist-test-slave-n3fx"
server_key: "c7eea93d25764d2214f29b15a90ceb87"
server_key_iv: "1da691d35b009c2125e4c51f3474c63f"
server_key_version: "encryptionkey@0"
I20260620 19:10:17.563557 13896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/minicluster-data/master-0/wal/instance:
uuid: "815622646e3045c889d0f94cd715cade"
format_stamp: "Formatted at 2026-06-20 19:10:17 on dist-test-slave-n3fx"
server_key: "c7eea93d25764d2214f29b15a90ceb87"
server_key_iv: "1da691d35b009c2125e4c51f3474c63f"
server_key_version: "encryptionkey@0"
I20260620 19:10:17.567229 13896 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 19:10:17.569631 13910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:17.570967 13896 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 19:10:17.571103 13896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/minicluster-data/master-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/minicluster-data/master-0/wal
uuid: "815622646e3045c889d0f94cd715cade"
format_stamp: "Formatted at 2026-06-20 19:10:17 on dist-test-slave-n3fx"
server_key: "c7eea93d25764d2214f29b15a90ceb87"
server_key_iv: "1da691d35b009c2125e4c51f3474c63f"
server_key_version: "encryptionkey@0"
I20260620 19:10:17.571205 13896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:17.587819 13896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:17.591130 13896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:17.591331 13896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:17.598248 13896 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.126:38073
I20260620 19:10:17.598534 13962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.126:38073 every 8 connection(s)
I20260620 19:10:17.598932 13896 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
I20260620 19:10:17.601729 13963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:17.607434 13963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 815622646e3045c889d0f94cd715cade: Bootstrap starting.
I20260620 19:10:17.609087 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 13896
I20260620 19:10:17.609217 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/minicluster-data/master-0/wal/instance
I20260620 19:10:17.609506 9837 external_mini_cluster.cc:1428] Setting key edc483170f5c67083ed8b13f8326c1ad
I20260620 19:10:17.610491 13963 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 815622646e3045c889d0f94cd715cade: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:17.611402 13963 log.cc:824] T 00000000000000000000000000000000 P 815622646e3045c889d0f94cd715cade: Log is configured to *not* fsync() on all Append() calls
I20260620 19:10:17.613739 13963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 815622646e3045c889d0f94cd715cade: No bootstrap required, opened a new log
I20260620 19:10:17.617964 13963 raft_consensus.cc:348] T 00000000000000000000000000000000 P 815622646e3045c889d0f94cd715cade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "815622646e3045c889d0f94cd715cade" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 38073 } }
I20260620 19:10:17.618259 13963 raft_consensus.cc:374] T 00000000000000000000000000000000 P 815622646e3045c889d0f94cd715cade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:17.618330 13963 raft_consensus.cc:729] T 00000000000000000000000000000000 P 815622646e3045c889d0f94cd715cade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 815622646e3045c889d0f94cd715cade, State: Initialized, Role: FOLLOWER
Jun 20 19:10:17 dist-test-slave-n3fx krb5kdc[13883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:10:17.618803 13963 consensus_queue.cc:260] T 00000000000000000000000000000000 P 815622646e3045c889d0f94cd715cade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "815622646e3045c889d0f94cd715cade" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 38073 } }
I20260620 19:10:17.618934 13963 raft_consensus.cc:388] T 00000000000000000000000000000000 P 815622646e3045c889d0f94cd715cade [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:10:17.618997 13963 raft_consensus.cc:482] T 00000000000000000000000000000000 P 815622646e3045c889d0f94cd715cade [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:10:17.619087 13963 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 815622646e3045c889d0f94cd715cade [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:17.620101 13963 raft_consensus.cc:504] T 00000000000000000000000000000000 P 815622646e3045c889d0f94cd715cade [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "815622646e3045c889d0f94cd715cade" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 38073 } }
I20260620 19:10:17.620496 13963 leader_election.cc:302] T 00000000000000000000000000000000 P 815622646e3045c889d0f94cd715cade [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 815622646e3045c889d0f94cd715cade; no voters:
I20260620 19:10:17.620819 13963 leader_election.cc:288] T 00000000000000000000000000000000 P 815622646e3045c889d0f94cd715cade [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:10:17.620913 13968 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 815622646e3045c889d0f94cd715cade [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:10:17.621070 13968 raft_consensus.cc:686] T 00000000000000000000000000000000 P 815622646e3045c889d0f94cd715cade [term 1 LEADER]: Becoming Leader. State: Replica: 815622646e3045c889d0f94cd715cade, State: Running, Role: LEADER
I20260620 19:10:17.621439 13968 consensus_queue.cc:237] T 00000000000000000000000000000000 P 815622646e3045c889d0f94cd715cade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "815622646e3045c889d0f94cd715cade" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 38073 } }
I20260620 19:10:17.622156 13963 sys_catalog.cc:564] T 00000000000000000000000000000000 P 815622646e3045c889d0f94cd715cade [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:10:17.623100 13970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 815622646e3045c889d0f94cd715cade [sys.catalog]: SysCatalogTable state changed. Reason: New leader 815622646e3045c889d0f94cd715cade. Latest consensus state: current_term: 1 leader_uuid: "815622646e3045c889d0f94cd715cade" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "815622646e3045c889d0f94cd715cade" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 38073 } } }
I20260620 19:10:17.623229 13970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 815622646e3045c889d0f94cd715cade [sys.catalog]: This master's current role is: LEADER
I20260620 19:10:17.623662 13969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 815622646e3045c889d0f94cd715cade [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "815622646e3045c889d0f94cd715cade" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "815622646e3045c889d0f94cd715cade" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 38073 } } }
I20260620 19:10:17.623757 13969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 815622646e3045c889d0f94cd715cade [sys.catalog]: This master's current role is: LEADER
I20260620 19:10:17.623759 13977 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:10:17.625525 13966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:17.610766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44100 (local address 127.9.155.126:38073)
0620 19:10:17.611202 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:17.611218 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:17.611256 (+ 38us) server_negotiation.cc:407] Connection header received
0620 19:10:17.612098 (+ 842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:17.612125 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:17.612539 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:17.612835 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:17.613737 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:17.614777 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:17.615940 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:17.616486 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:17.616604 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:17.619257 (+ 2653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:17.619262 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:17.619275 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:17.619300 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:17.621236 (+ 1936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:17.621736 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:17.621740 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:17.621745 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:17.621866 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:17.622152 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:17.622156 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:17.622158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:17.622780 (+ 622us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:17.624070 (+ 1290us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:17.624250 (+ 180us) server_negotiation.cc:299] Negotiation successful
0620 19:10:17.624574 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":119,"threads_started":1}
I20260620 19:10:17.626974 13977 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:10:17.632897 13977 catalog_manager.cc:1347] Generated new cluster ID: 11a405a4cfa941baaa3d09743449e8e0
I20260620 19:10:17.632995 13977 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:10:17.642942 13977 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:10:17.643947 13977 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:10:17.648830 13977 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 815622646e3045c889d0f94cd715cade: Generated new TSK 0
I20260620 19:10:17.649613 13977 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.155.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.155.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.155.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.65.keytab.
Entry for principal kudu/127.9.155.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.65.keytab.
I20260620 19:10:17.679647 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.65:0
--local_ip_for_outbound_sockets=127.9.155.65
--webserver_interface=127.9.155.65
--webserver_port=0
--tserver_master_addrs=127.9.155.126:38073
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.65.keytab
--principal=kudu/127.9.155.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:36915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:10:17.789052 13989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:17.789310 13989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:17.789359 13989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:17.793411 13989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:17.793474 13989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:17.793540 13989 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.65
I20260620 19:10:17.799021 13989 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:36915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.65.keytab
--principal=kudu/127.9.155.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.155.65
--webserver_port=0
--tserver_master_addrs=127.9.155.126:38073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13989
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.65
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:10:17.800100 13989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:17.801102 13989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:10:17.811944 13995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:17.811944 13994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:17.811982 13997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:17.812250 13989 server_base.cc:1034] running on GCE node
I20260620 19:10:17.812769 13989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:17.813403 13989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:17.814566 13989 hybrid_clock.cc:648] HybridClock initialized: now 1781982617814539 us; error 42 us; skew 500 ppm
Jun 20 19:10:17 dist-test-slave-n3fx krb5kdc[13883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982617, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:17.817214 13989 init.cc:377] Logged in from keytab as kudu/127.9.155.65@KRBTEST.COM (short username kudu)
I20260620 19:10:17.817911 13989 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:17.817982 13989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:17.818152 13989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:10:17.820125 13989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/minicluster-data/ts-0/data/instance:
uuid: "367ccb48ccc64cda90859fb80a2e3b58"
format_stamp: "Formatted at 2026-06-20 19:10:17 on dist-test-slave-n3fx"
server_key: "235102f4880bf27c7c91b0e2d0ba3d93"
server_key_iv: "dc3f4788d038d88dffaea87534163dab"
server_key_version: "encryptionkey@0"
I20260620 19:10:17.820668 13989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/minicluster-data/ts-0/wal/instance:
uuid: "367ccb48ccc64cda90859fb80a2e3b58"
format_stamp: "Formatted at 2026-06-20 19:10:17 on dist-test-slave-n3fx"
server_key: "235102f4880bf27c7c91b0e2d0ba3d93"
server_key_iv: "dc3f4788d038d88dffaea87534163dab"
server_key_version: "encryptionkey@0"
I20260620 19:10:17.824739 13989 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 19:10:17.827181 14003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:17.828544 13989 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:10:17.828648 13989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/minicluster-data/ts-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/minicluster-data/ts-0/wal
uuid: "367ccb48ccc64cda90859fb80a2e3b58"
format_stamp: "Formatted at 2026-06-20 19:10:17 on dist-test-slave-n3fx"
server_key: "235102f4880bf27c7c91b0e2d0ba3d93"
server_key_iv: "dc3f4788d038d88dffaea87534163dab"
server_key_version: "encryptionkey@0"
I20260620 19:10:17.828759 13989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:17.844370 13989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:17.847939 13989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:17.848137 13989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:17.848765 13989 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:10:17.849613 13989 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:10:17.849658 13989 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:17.849701 13989 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:10:17.849731 13989 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:17.858268 13989 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.65:36057
I20260620 19:10:17.858287 14116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.65:36057 every 8 connection(s)
I20260620 19:10:17.858909 13989 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
I20260620 19:10:17.868105 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 13989
I20260620 19:10:17.868258 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/minicluster-data/ts-0/wal/instance
Jun 20 19:10:17 dist-test-slave-n3fx krb5kdc[13883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982617, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.65@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:10:17.868590 9837 external_mini_cluster.cc:1428] Setting key 097b28dea221d85656bb9ac8fa9017b9
I20260620 19:10:17.872920 13966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:17.860772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.65:40675 (local address 127.9.155.126:38073)
0620 19:10:17.860935 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:17.860939 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:17.861761 (+ 822us) server_negotiation.cc:407] Connection header received
0620 19:10:17.863031 (+ 1270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:17.863035 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:17.863092 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:17.863166 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:17.864397 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:17.864931 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:17.866051 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:17.866299 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:17.866390 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:17.869073 (+ 2683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:17.869077 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:17.869079 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:17.869105 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:17.870722 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:17.871268 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:17.871271 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:17.871273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:17.871360 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:17.871757 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:17.871761 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:17.871762 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:17.871956 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:17.872045 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:17.872624 (+ 579us) server_negotiation.cc:299] Negotiation successful
0620 19:10:17.872756 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 19:10:17.873807 14119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:17.861099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:38073 (local address 127.9.155.65:40675)
0620 19:10:17.861597 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:17.861642 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:17.862806 (+ 1164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:17.863347 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:17.863355 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:17.863718 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:17.864251 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:17.864271 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:17.865053 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:17.865056 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:17.865930 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:17.865939 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:17.866449 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:17.866454 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:17.866611 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:17.867211 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:17.867228 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:17.868877 (+ 1649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:17.870859 (+ 1982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:17.870865 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:17.870903 (+ 38us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:17.871176 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:17.871495 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:17.871497 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:17.871499 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:17.871663 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:17.872062 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:17.872070 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:17.872393 (+ 323us) client_negotiation.cc:769] Sending connection context
0620 19:10:17.872571 (+ 178us) client_negotiation.cc:240] Negotiation successful
0620 19:10:17.872935 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":123,"threads_started":1}
I20260620 19:10:17.875319 14117 heartbeater.cc:347] Connected to a master server at 127.9.155.126:38073
I20260620 19:10:17.875586 14117 heartbeater.cc:464] Registering TS with master...
I20260620 19:10:17.876156 14117 heartbeater.cc:511] Master 127.9.155.126:38073 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.9.155.66@KRBTEST.COM; defaulting to no policy
I20260620 19:10:17.877962 13927 ts_manager.cc:194] Registered new tserver with Master: 367ccb48ccc64cda90859fb80a2e3b58 (127.9.155.65:36057)
I20260620 19:10:17.879150 13927 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.155.65@KRBTEST.COM'} at 127.9.155.65:40675
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.155.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.155.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.66.keytab.
Entry for principal kudu/127.9.155.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.66.keytab.
I20260620 19:10:17.895946 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.66:0
--local_ip_for_outbound_sockets=127.9.155.66
--webserver_interface=127.9.155.66
--webserver_port=0
--tserver_master_addrs=127.9.155.126:38073
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.66.keytab
--principal=kudu/127.9.155.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:36915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:10:18.008069 14122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:18.008392 14122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:18.008440 14122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:18.012609 14122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:18.012676 14122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:18.012738 14122 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.66
I20260620 19:10:18.018059 14122 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:36915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.66.keytab
--principal=kudu/127.9.155.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.155.66
--webserver_port=0
--tserver_master_addrs=127.9.155.126:38073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14122
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.66
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:10:18.019080 14122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:18.020066 14122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 19:10:18.030748 14122 server_base.cc:1034] running on GCE node
W20260620 19:10:18.030689 14130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:18.030689 14127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:18.030689 14128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:18.031643 14122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:18.032392 14122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:18.033629 14122 hybrid_clock.cc:648] HybridClock initialized: now 1781982618033621 us; error 41 us; skew 500 ppm
Jun 20 19:10:18 dist-test-slave-n3fx krb5kdc[13883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982618, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:18.036900 14122 init.cc:377] Logged in from keytab as kudu/127.9.155.66@KRBTEST.COM (short username kudu)
I20260620 19:10:18.037720 14122 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:18.037796 14122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:18.038151 14122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:10:18.040499 14122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/minicluster-data/ts-1/data/instance:
uuid: "e56807c1c3f542e7b8a10080b1de51b9"
format_stamp: "Formatted at 2026-06-20 19:10:18 on dist-test-slave-n3fx"
server_key: "ed3dc4ffefddaa0aac834bf9f1a150a9"
server_key_iv: "d86b86d4bc45c6a30d171bd48fb60aad"
server_key_version: "encryptionkey@0"
I20260620 19:10:18.041091 14122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/minicluster-data/ts-1/wal/instance:
uuid: "e56807c1c3f542e7b8a10080b1de51b9"
format_stamp: "Formatted at 2026-06-20 19:10:18 on dist-test-slave-n3fx"
server_key: "ed3dc4ffefddaa0aac834bf9f1a150a9"
server_key_iv: "d86b86d4bc45c6a30d171bd48fb60aad"
server_key_version: "encryptionkey@0"
I20260620 19:10:18.045338 14122 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 19:10:18.048529 14136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:18.050411 14122 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 19:10:18.050575 14122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/minicluster-data/ts-1/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/minicluster-data/ts-1/wal
uuid: "e56807c1c3f542e7b8a10080b1de51b9"
format_stamp: "Formatted at 2026-06-20 19:10:18 on dist-test-slave-n3fx"
server_key: "ed3dc4ffefddaa0aac834bf9f1a150a9"
server_key_iv: "d86b86d4bc45c6a30d171bd48fb60aad"
server_key_version: "encryptionkey@0"
I20260620 19:10:18.050711 14122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:18.070169 14122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:18.073920 14122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:18.074106 14122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:18.074721 14122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:10:18.075626 14122 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:10:18.075675 14122 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:18.075726 14122 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:10:18.075757 14122 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:18.084131 14122 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.66:37245
I20260620 19:10:18.084149 14249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.66:37245 every 8 connection(s)
I20260620 19:10:18.084795 14122 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
I20260620 19:10:18.092911 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 14122
I20260620 19:10:18.093053 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/minicluster-data/ts-1/wal/instance
I20260620 19:10:18.093403 9837 external_mini_cluster.cc:1428] Setting key c717eed5c5f7802086a961d3db8b7a83
Jun 20 19:10:18 dist-test-slave-n3fx krb5kdc[13883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982618, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.66@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:10:18.098408 13966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:18.087011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.66:57011 (local address 127.9.155.126:38073)
0620 19:10:18.087139 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:18.087142 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:18.087541 (+ 399us) server_negotiation.cc:407] Connection header received
0620 19:10:18.088927 (+ 1386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:18.088931 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:18.088987 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:18.089070 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:18.090174 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:18.090690 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:18.091806 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:18.092035 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:18.092114 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:18.094870 (+ 2756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:18.094873 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:18.094875 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:18.094897 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:18.096341 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:18.096893 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:18.096897 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:18.096899 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:18.096997 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:18.097305 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:18.097308 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:18.097310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:18.097530 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:18.097633 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:18.098118 (+ 485us) server_negotiation.cc:299] Negotiation successful
0620 19:10:18.098250 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 19:10:18.099227 14252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:18.086906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:38073 (local address 127.9.155.66:57011)
0620 19:10:18.087397 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:18.087449 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:18.088713 (+ 1264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:18.089210 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:18.089217 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:18.089580 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:18.090047 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:18.090061 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:18.090791 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:18.090795 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:18.091685 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:18.091694 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:18.092165 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:18.092168 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:18.092427 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:18.092986 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:18.093007 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:18.094710 (+ 1703us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:18.096482 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:18.096488 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:18.096503 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:18.096790 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:18.097087 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:18.097089 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:18.097091 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:18.097232 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:18.097620 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:18.097625 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:18.097875 (+ 250us) client_negotiation.cc:769] Sending connection context
0620 19:10:18.098043 (+ 168us) client_negotiation.cc:240] Negotiation successful
0620 19:10:18.098309 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":111,"threads_started":1}
I20260620 19:10:18.100410 14250 heartbeater.cc:347] Connected to a master server at 127.9.155.126:38073
I20260620 19:10:18.100682 14250 heartbeater.cc:464] Registering TS with master...
I20260620 19:10:18.101235 14250 heartbeater.cc:511] Master 127.9.155.126:38073 requested a full tablet report, sending...
I20260620 19:10:18.102425 13927 ts_manager.cc:194] Registered new tserver with Master: e56807c1c3f542e7b8a10080b1de51b9 (127.9.155.66:37245)
I20260620 19:10:18.103010 13927 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.155.66@KRBTEST.COM'} at 127.9.155.66:57011
WARNING: no policy specified for kudu/127.9.155.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.155.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.155.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.67.keytab.
Entry for principal kudu/127.9.155.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.67.keytab.
I20260620 19:10:18.123378 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.67:0
--local_ip_for_outbound_sockets=127.9.155.67
--webserver_interface=127.9.155.67
--webserver_port=0
--tserver_master_addrs=127.9.155.126:38073
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.67.keytab
--principal=kudu/127.9.155.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:36915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:10:18.237354 14255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:18.237650 14255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:18.237697 14255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:18.242074 14255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:18.242159 14255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:18.242224 14255 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.67
I20260620 19:10:18.247815 14255 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:36915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.67.keytab
--principal=kudu/127.9.155.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.155.67
--webserver_port=0
--tserver_master_addrs=127.9.155.126:38073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14255
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.67
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:10:18.249022 14255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:18.250069 14255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:10:18.259810 14263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:18.259810 14260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:18.259879 14255 server_base.cc:1034] running on GCE node
W20260620 19:10:18.259810 14261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:18.260478 14255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:18.261111 14255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:18.262260 14255 hybrid_clock.cc:648] HybridClock initialized: now 1781982618262235 us; error 40 us; skew 500 ppm
Jun 20 19:10:18 dist-test-slave-n3fx krb5kdc[13883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982618, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:18.265053 14255 init.cc:377] Logged in from keytab as kudu/127.9.155.67@KRBTEST.COM (short username kudu)
I20260620 19:10:18.265769 14255 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:18.265836 14255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:18.266021 14255 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:10:18.268103 14255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/minicluster-data/ts-2/data/instance:
uuid: "fad2ed6f21704cafbabf09532815df47"
format_stamp: "Formatted at 2026-06-20 19:10:18 on dist-test-slave-n3fx"
server_key: "cced19ffaa344173de3ecb9493183af4"
server_key_iv: "ef58bea4c200e67fc8a3ffe53da0e339"
server_key_version: "encryptionkey@0"
I20260620 19:10:18.268656 14255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/minicluster-data/ts-2/wal/instance:
uuid: "fad2ed6f21704cafbabf09532815df47"
format_stamp: "Formatted at 2026-06-20 19:10:18 on dist-test-slave-n3fx"
server_key: "cced19ffaa344173de3ecb9493183af4"
server_key_iv: "ef58bea4c200e67fc8a3ffe53da0e339"
server_key_version: "encryptionkey@0"
I20260620 19:10:18.272502 14255 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 19:10:18.275095 14269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:18.276881 14255 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:10:18.277017 14255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/minicluster-data/ts-2/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/minicluster-data/ts-2/wal
uuid: "fad2ed6f21704cafbabf09532815df47"
format_stamp: "Formatted at 2026-06-20 19:10:18 on dist-test-slave-n3fx"
server_key: "cced19ffaa344173de3ecb9493183af4"
server_key_iv: "ef58bea4c200e67fc8a3ffe53da0e339"
server_key_version: "encryptionkey@0"
I20260620 19:10:18.277140 14255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:18.302274 14255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:18.305748 14255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:18.305955 14255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:18.306577 14255 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:10:18.307534 14255 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:10:18.307605 14255 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:18.307665 14255 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:10:18.307695 14255 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:18.318204 14255 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.67:35437
I20260620 19:10:18.319164 14255 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
I20260620 19:10:18.319996 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 14255
I20260620 19:10:18.320101 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/minicluster-data/ts-2/wal/instance
I20260620 19:10:18.320437 9837 external_mini_cluster.cc:1428] Setting key e6c733d5801e6b59f414e1beb93210de
I20260620 19:10:18.320413 14382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.67:35437 every 8 connection(s)
Jun 20 19:10:18 dist-test-slave-n3fx krb5kdc[13883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982618, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.67@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:10:18.339545 13966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:18.322942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.67:57645 (local address 127.9.155.126:38073)
0620 19:10:18.323522 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:18.323527 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:18.323546 (+ 19us) server_negotiation.cc:407] Connection header received
0620 19:10:18.325346 (+ 1800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:18.325350 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:18.325408 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:18.325506 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:18.328158 (+ 2652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:18.329140 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:18.330998 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:18.331453 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:18.331574 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:18.334924 (+ 3350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:18.334928 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:18.334931 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:18.334959 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:18.336692 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:18.337468 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:18.337472 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:18.337474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:18.337575 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:18.338045 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:18.338050 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:18.338052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:18.338313 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:18.338388 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:18.339215 (+ 827us) server_negotiation.cc:299] Negotiation successful
0620 19:10:18.339366 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472}
I20260620 19:10:18.340629 14385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:18.322657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:38073 (local address 127.9.155.67:57645)
0620 19:10:18.323356 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:18.323419 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:18.325087 (+ 1668us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:18.326053 (+ 966us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:18.326178 (+ 125us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:18.327126 (+ 948us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:18.327951 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:18.327979 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:18.329313 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:18.329317 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:18.330823 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:18.330838 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:18.331646 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:18.331651 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:18.331886 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:18.332665 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:18.332686 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:18.334674 (+ 1988us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:18.336869 (+ 2195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:18.336878 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:18.336898 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:18.337333 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:18.337698 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:18.337701 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:18.337704 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:18.337926 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:18.338477 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:18.338484 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:18.338891 (+ 407us) client_negotiation.cc:769] Sending connection context
0620 19:10:18.339132 (+ 241us) client_negotiation.cc:240] Negotiation successful
0620 19:10:18.339546 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":184,"threads_started":1}
I20260620 19:10:18.342257 14383 heartbeater.cc:347] Connected to a master server at 127.9.155.126:38073
I20260620 19:10:18.342537 14383 heartbeater.cc:464] Registering TS with master...
I20260620 19:10:18.343116 14383 heartbeater.cc:511] Master 127.9.155.126:38073 requested a full tablet report, sending...
I20260620 19:10:18.344431 13927 ts_manager.cc:194] Registered new tserver with Master: fad2ed6f21704cafbabf09532815df47 (127.9.155.67:35437)
I20260620 19:10:18.345011 13927 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.155.67@KRBTEST.COM'} at 127.9.155.67:57645
I20260620 19:10:18.350401 9837 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:10:18.360661 13966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:18.352628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58580 (local address 127.9.155.126:38073)
0620 19:10:18.352803 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:18.352807 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:18.352898 (+ 91us) server_negotiation.cc:407] Connection header received
0620 19:10:18.353049 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:18.353052 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:18.353099 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:18.353170 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:18.353910 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:18.354431 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:18.355381 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:18.355608 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:18.355700 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:18.357135 (+ 1435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:18.357139 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:18.357141 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:18.357168 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:18.358672 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:18.359155 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:18.359158 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:18.359159 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:18.359247 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:18.359577 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:18.359581 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:18.359582 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:18.359905 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:18.360027 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:18.360393 (+ 366us) server_negotiation.cc:299] Negotiation successful
0620 19:10:18.360505 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 19:10:18.364260 13927 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58580:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:10:18.367372 13927 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:10:18.383374 14398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:18.377269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.66:37245 (local address 127.0.0.1:45944)
0620 19:10:18.378224 (+ 955us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:18.378279 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:18.378446 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:18.378984 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:18.378990 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:18.379013 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:10:18.379148 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:18.379154 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:18.380656 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:18.380659 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:18.381982 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:18.381990 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:18.382793 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:18.382796 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:18.382929 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:18.382970 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 19:10:18.383078 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 19:10:18.383211 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":773,"thread_start_us":78,"threads_started":1}
I20260620 19:10:18.383380 14396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:18.377554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45944 (local address 127.9.155.66:37245)
0620 19:10:18.378118 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:18.378130 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:18.378364 (+ 234us) server_negotiation.cc:407] Connection header received
0620 19:10:18.378608 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:18.378613 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:18.378832 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:18.378994 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:10:18.379274 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:18.380477 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:18.382094 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:18.382675 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:18.382776 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:18.382937 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:18.383125 (+ 188us) server_negotiation.cc:299] Negotiation successful
0620 19:10:18.383228 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":137,"threads_started":1}
I20260620 19:10:18.383855 14399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:18.377996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39294 (local address 127.9.155.67:35437)
0620 19:10:18.378435 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:18.378446 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:18.378777 (+ 331us) server_negotiation.cc:407] Connection header received
0620 19:10:18.378856 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:18.378861 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:18.379041 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:18.379182 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:10:18.379416 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:18.380249 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:18.381944 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:18.382802 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:18.382908 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:18.383040 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:18.383610 (+ 570us) server_negotiation.cc:299] Negotiation successful
0620 19:10:18.383710 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":136,"threads_started":1}
I20260620 19:10:18.383896 14394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:18.377277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.65:36057 (local address 127.0.0.1:54138)
0620 19:10:18.378830 (+ 1553us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:18.378843 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:18.378914 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:18.379404 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:18.379407 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:18.379425 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:10:18.379496 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:18.379501 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:18.380787 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:18.380790 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:18.381866 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:18.381877 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:18.383101 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:18.383105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:18.383262 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:18.383286 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 19:10:18.383354 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 19:10:18.383429 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1481,"mutex_wait_us":618,"thread_start_us":100,"threads_started":1}
I20260620 19:10:18.384045 14397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:18.377654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54138 (local address 127.9.155.65:36057)
0620 19:10:18.378294 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:18.378300 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:18.378998 (+ 698us) server_negotiation.cc:407] Connection header received
0620 19:10:18.379070 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:18.379075 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:18.379274 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:18.379406 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:10:18.379602 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:18.380682 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:18.382000 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:18.382809 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:18.382908 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:18.383040 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:18.383795 (+ 755us) server_negotiation.cc:299] Negotiation successful
0620 19:10:18.383914 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":85,"threads_started":1}
I20260620 19:10:18.384164 14395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:18.377462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.67:35437 (local address 127.0.0.1:39294)
0620 19:10:18.378583 (+ 1121us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:18.378596 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:18.378687 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:18.379177 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:18.379180 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:18.379197 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:10:18.379296 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:18.379303 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:18.380406 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:18.380410 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:18.381838 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:18.381849 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:18.383101 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:18.383104 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:18.383340 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:18.383360 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 19:10:18.383415 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 19:10:18.383471 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1049,"mutex_wait_us":58,"thread_start_us":102,"threads_started":1}
I20260620 19:10:18.386669 14051 tablet_service.cc:1467] Processing CreateTablet for tablet 89f4c1af687845b4a96759b172cd6aa6 (DEFAULT_TABLE table=test-table [id=d1a1f7d79a7d4f499c18d690f9f496ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:10:18.386699 14184 tablet_service.cc:1467] Processing CreateTablet for tablet 89f4c1af687845b4a96759b172cd6aa6 (DEFAULT_TABLE table=test-table [id=d1a1f7d79a7d4f499c18d690f9f496ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:10:18.386950 14317 tablet_service.cc:1467] Processing CreateTablet for tablet 89f4c1af687845b4a96759b172cd6aa6 (DEFAULT_TABLE table=test-table [id=d1a1f7d79a7d4f499c18d690f9f496ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:10:18.387612 14051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89f4c1af687845b4a96759b172cd6aa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:18.387729 14184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89f4c1af687845b4a96759b172cd6aa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:18.387764 14317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89f4c1af687845b4a96759b172cd6aa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:18.394449 14400 tablet_bootstrap.cc:492] T 89f4c1af687845b4a96759b172cd6aa6 P 367ccb48ccc64cda90859fb80a2e3b58: Bootstrap starting.
I20260620 19:10:18.395512 14401 tablet_bootstrap.cc:492] T 89f4c1af687845b4a96759b172cd6aa6 P e56807c1c3f542e7b8a10080b1de51b9: Bootstrap starting.
I20260620 19:10:18.396025 14402 tablet_bootstrap.cc:492] T 89f4c1af687845b4a96759b172cd6aa6 P fad2ed6f21704cafbabf09532815df47: Bootstrap starting.
I20260620 19:10:18.396643 14400 tablet_bootstrap.cc:654] T 89f4c1af687845b4a96759b172cd6aa6 P 367ccb48ccc64cda90859fb80a2e3b58: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:18.397339 14400 log.cc:824] T 89f4c1af687845b4a96759b172cd6aa6 P 367ccb48ccc64cda90859fb80a2e3b58: Log is configured to *not* fsync() on all Append() calls
I20260620 19:10:18.397632 14401 tablet_bootstrap.cc:654] T 89f4c1af687845b4a96759b172cd6aa6 P e56807c1c3f542e7b8a10080b1de51b9: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:18.398034 14402 tablet_bootstrap.cc:654] T 89f4c1af687845b4a96759b172cd6aa6 P fad2ed6f21704cafbabf09532815df47: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:18.398900 14401 log.cc:824] T 89f4c1af687845b4a96759b172cd6aa6 P e56807c1c3f542e7b8a10080b1de51b9: Log is configured to *not* fsync() on all Append() calls
I20260620 19:10:18.399222 14400 tablet_bootstrap.cc:492] T 89f4c1af687845b4a96759b172cd6aa6 P 367ccb48ccc64cda90859fb80a2e3b58: No bootstrap required, opened a new log
I20260620 19:10:18.399194 14402 log.cc:824] T 89f4c1af687845b4a96759b172cd6aa6 P fad2ed6f21704cafbabf09532815df47: Log is configured to *not* fsync() on all Append() calls
I20260620 19:10:18.399434 14400 ts_tablet_manager.cc:1397] T 89f4c1af687845b4a96759b172cd6aa6 P 367ccb48ccc64cda90859fb80a2e3b58: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 19:10:18.401103 14401 tablet_bootstrap.cc:492] T 89f4c1af687845b4a96759b172cd6aa6 P e56807c1c3f542e7b8a10080b1de51b9: No bootstrap required, opened a new log
I20260620 19:10:18.401296 14401 ts_tablet_manager.cc:1397] T 89f4c1af687845b4a96759b172cd6aa6 P e56807c1c3f542e7b8a10080b1de51b9: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 19:10:18.402019 14402 tablet_bootstrap.cc:492] T 89f4c1af687845b4a96759b172cd6aa6 P fad2ed6f21704cafbabf09532815df47: No bootstrap required, opened a new log
I20260620 19:10:18.402200 14402 ts_tablet_manager.cc:1397] T 89f4c1af687845b4a96759b172cd6aa6 P fad2ed6f21704cafbabf09532815df47: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 19:10:18.402999 14400 raft_consensus.cc:348] T 89f4c1af687845b4a96759b172cd6aa6 P 367ccb48ccc64cda90859fb80a2e3b58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56807c1c3f542e7b8a10080b1de51b9" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 37245 } } peers { permanent_uuid: "367ccb48ccc64cda90859fb80a2e3b58" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 36057 } } peers { permanent_uuid: "fad2ed6f21704cafbabf09532815df47" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 35437 } }
I20260620 19:10:18.403234 14400 raft_consensus.cc:374] T 89f4c1af687845b4a96759b172cd6aa6 P 367ccb48ccc64cda90859fb80a2e3b58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:18.403285 14400 raft_consensus.cc:729] T 89f4c1af687845b4a96759b172cd6aa6 P 367ccb48ccc64cda90859fb80a2e3b58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 367ccb48ccc64cda90859fb80a2e3b58, State: Initialized, Role: FOLLOWER
I20260620 19:10:18.403767 14400 consensus_queue.cc:260] T 89f4c1af687845b4a96759b172cd6aa6 P 367ccb48ccc64cda90859fb80a2e3b58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56807c1c3f542e7b8a10080b1de51b9" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 37245 } } peers { permanent_uuid: "367ccb48ccc64cda90859fb80a2e3b58" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 36057 } } peers { permanent_uuid: "fad2ed6f21704cafbabf09532815df47" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 35437 } }
I20260620 19:10:18.404469 14117 heartbeater.cc:503] Master 127.9.155.126:38073 was elected leader, sending a full tablet report...
I20260620 19:10:18.404830 14400 ts_tablet_manager.cc:1428] T 89f4c1af687845b4a96759b172cd6aa6 P 367ccb48ccc64cda90859fb80a2e3b58: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 19:10:18.405160 14401 raft_consensus.cc:348] T 89f4c1af687845b4a96759b172cd6aa6 P e56807c1c3f542e7b8a10080b1de51b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56807c1c3f542e7b8a10080b1de51b9" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 37245 } } peers { permanent_uuid: "367ccb48ccc64cda90859fb80a2e3b58" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 36057 } } peers { permanent_uuid: "fad2ed6f21704cafbabf09532815df47" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 35437 } }
I20260620 19:10:18.405457 14401 raft_consensus.cc:374] T 89f4c1af687845b4a96759b172cd6aa6 P e56807c1c3f542e7b8a10080b1de51b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:18.405516 14401 raft_consensus.cc:729] T 89f4c1af687845b4a96759b172cd6aa6 P e56807c1c3f542e7b8a10080b1de51b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e56807c1c3f542e7b8a10080b1de51b9, State: Initialized, Role: FOLLOWER
I20260620 19:10:18.405928 14402 raft_consensus.cc:348] T 89f4c1af687845b4a96759b172cd6aa6 P fad2ed6f21704cafbabf09532815df47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56807c1c3f542e7b8a10080b1de51b9" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 37245 } } peers { permanent_uuid: "367ccb48ccc64cda90859fb80a2e3b58" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 36057 } } peers { permanent_uuid: "fad2ed6f21704cafbabf09532815df47" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 35437 } }
I20260620 19:10:18.406080 14401 consensus_queue.cc:260] T 89f4c1af687845b4a96759b172cd6aa6 P e56807c1c3f542e7b8a10080b1de51b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56807c1c3f542e7b8a10080b1de51b9" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 37245 } } peers { permanent_uuid: "367ccb48ccc64cda90859fb80a2e3b58" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 36057 } } peers { permanent_uuid: "fad2ed6f21704cafbabf09532815df47" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 35437 } }
I20260620 19:10:18.406213 14402 raft_consensus.cc:374] T 89f4c1af687845b4a96759b172cd6aa6 P fad2ed6f21704cafbabf09532815df47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:18.406298 14402 raft_consensus.cc:729] T 89f4c1af687845b4a96759b172cd6aa6 P fad2ed6f21704cafbabf09532815df47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fad2ed6f21704cafbabf09532815df47, State: Initialized, Role: FOLLOWER
I20260620 19:10:18.406808 14250 heartbeater.cc:503] Master 127.9.155.126:38073 was elected leader, sending a full tablet report...
I20260620 19:10:18.407164 14401 ts_tablet_manager.cc:1428] T 89f4c1af687845b4a96759b172cd6aa6 P e56807c1c3f542e7b8a10080b1de51b9: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 19:10:18.407747 14402 consensus_queue.cc:260] T 89f4c1af687845b4a96759b172cd6aa6 P fad2ed6f21704cafbabf09532815df47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56807c1c3f542e7b8a10080b1de51b9" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 37245 } } peers { permanent_uuid: "367ccb48ccc64cda90859fb80a2e3b58" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 36057 } } peers { permanent_uuid: "fad2ed6f21704cafbabf09532815df47" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 35437 } }
I20260620 19:10:18.408550 14383 heartbeater.cc:503] Master 127.9.155.126:38073 was elected leader, sending a full tablet report...
I20260620 19:10:18.408901 14402 ts_tablet_manager.cc:1428] T 89f4c1af687845b4a96759b172cd6aa6 P fad2ed6f21704cafbabf09532815df47: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260620 19:10:18.570570 14384 tablet.cc:2369] T 89f4c1af687845b4a96759b172cd6aa6 P fad2ed6f21704cafbabf09532815df47: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:10:18.583496 14408 raft_consensus.cc:482] T 89f4c1af687845b4a96759b172cd6aa6 P fad2ed6f21704cafbabf09532815df47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:10:18.583719 14408 raft_consensus.cc:504] T 89f4c1af687845b4a96759b172cd6aa6 P fad2ed6f21704cafbabf09532815df47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56807c1c3f542e7b8a10080b1de51b9" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 37245 } } peers { permanent_uuid: "367ccb48ccc64cda90859fb80a2e3b58" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 36057 } } peers { permanent_uuid: "fad2ed6f21704cafbabf09532815df47" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 35437 } }
I20260620 19:10:18.584941 14408 leader_election.cc:288] T 89f4c1af687845b4a96759b172cd6aa6 P fad2ed6f21704cafbabf09532815df47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e56807c1c3f542e7b8a10080b1de51b9 (127.9.155.66:37245), 367ccb48ccc64cda90859fb80a2e3b58 (127.9.155.65:36057)
W20260620 19:10:18.585799 14251 tablet.cc:2369] T 89f4c1af687845b4a96759b172cd6aa6 P e56807c1c3f542e7b8a10080b1de51b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:10:18.589181 14396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:18.585330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.67:45037 (local address 127.9.155.66:37245)
0620 19:10:18.585447 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:18.585452 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:18.585467 (+ 15us) server_negotiation.cc:407] Connection header received
0620 19:10:18.585605 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:18.585609 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:18.585659 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:18.585728 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:10:18.586084 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:18.586737 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:18.587977 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:18.588583 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:18.588678 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:18.588732 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:18.588994 (+ 262us) server_negotiation.cc:299] Negotiation successful
0620 19:10:18.589073 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 19:10:18.589184 14385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:18.585205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.66:37245 (local address 127.9.155.67:45037)
0620 19:10:18.585354 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:18.585370 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:18.585479 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:18.585768 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:18.585772 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:18.585796 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:10:18.585963 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:18.585970 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:18.586854 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:18.586857 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:18.587820 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:18.587831 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:18.588708 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:18.588712 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:18.588834 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:18.588861 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 19:10:18.588949 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 19:10:18.589038 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 19:10:18.589812 14397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:18.585330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.67:43031 (local address 127.9.155.65:36057)
0620 19:10:18.585455 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:18.585458 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:18.585902 (+ 444us) server_negotiation.cc:407] Connection header received
0620 19:10:18.586044 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:18.586047 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:18.586099 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:18.586144 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:10:18.586556 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:18.587196 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:18.588570 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:18.589167 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:18.589285 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:18.589345 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:18.589649 (+ 304us) server_negotiation.cc:299] Negotiation successful
0620 19:10:18.589719 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 19:10:18.589833 14409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:18.585205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.65:36057 (local address 127.9.155.67:43031)
0620 19:10:18.585804 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:18.585817 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:18.585909 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:18.586299 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:18.586310 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:18.586328 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:10:18.586435 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:18.586441 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:18.587337 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:18.587340 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:18.588410 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:18.588422 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:18.589386 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:18.589390 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:18.589504 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:18.589529 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 19:10:18.589615 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 19:10:18.589689 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"mutex_wait_us":260,"thread_start_us":119,"threads_started":1}
I20260620 19:10:18.589962 14204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89f4c1af687845b4a96759b172cd6aa6" candidate_uuid: "fad2ed6f21704cafbabf09532815df47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e56807c1c3f542e7b8a10080b1de51b9" is_pre_election: true
I20260620 19:10:18.590324 14204 raft_consensus.cc:2454] T 89f4c1af687845b4a96759b172cd6aa6 P e56807c1c3f542e7b8a10080b1de51b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fad2ed6f21704cafbabf09532815df47 in term 0.
I20260620 19:10:18.590509 14071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89f4c1af687845b4a96759b172cd6aa6" candidate_uuid: "fad2ed6f21704cafbabf09532815df47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "367ccb48ccc64cda90859fb80a2e3b58" is_pre_election: true
I20260620 19:10:18.590767 14071 raft_consensus.cc:2454] T 89f4c1af687845b4a96759b172cd6aa6 P 367ccb48ccc64cda90859fb80a2e3b58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fad2ed6f21704cafbabf09532815df47 in term 0.
I20260620 19:10:18.590799 14270 leader_election.cc:302] T 89f4c1af687845b4a96759b172cd6aa6 P fad2ed6f21704cafbabf09532815df47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e56807c1c3f542e7b8a10080b1de51b9, fad2ed6f21704cafbabf09532815df47; no voters:
I20260620 19:10:18.591080 14408 raft_consensus.cc:2781] T 89f4c1af687845b4a96759b172cd6aa6 P fad2ed6f21704cafbabf09532815df47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:10:18.591180 14408 raft_consensus.cc:482] T 89f4c1af687845b4a96759b172cd6aa6 P fad2ed6f21704cafbabf09532815df47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:10:18.591224 14408 raft_consensus.cc:3037] T 89f4c1af687845b4a96759b172cd6aa6 P fad2ed6f21704cafbabf09532815df47 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:18.592281 14408 raft_consensus.cc:504] T 89f4c1af687845b4a96759b172cd6aa6 P fad2ed6f21704cafbabf09532815df47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56807c1c3f542e7b8a10080b1de51b9" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 37245 } } peers { permanent_uuid: "367ccb48ccc64cda90859fb80a2e3b58" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 36057 } } peers { permanent_uuid: "fad2ed6f21704cafbabf09532815df47" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 35437 } }
I20260620 19:10:18.592608 14408 leader_election.cc:288] T 89f4c1af687845b4a96759b172cd6aa6 P fad2ed6f21704cafbabf09532815df47 [CANDIDATE]: Term 1 election: Requested vote from peers e56807c1c3f542e7b8a10080b1de51b9 (127.9.155.66:37245), 367ccb48ccc64cda90859fb80a2e3b58 (127.9.155.65:36057)
I20260620 19:10:18.592986 14071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89f4c1af687845b4a96759b172cd6aa6" candidate_uuid: "fad2ed6f21704cafbabf09532815df47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "367ccb48ccc64cda90859fb80a2e3b58"
I20260620 19:10:18.593104 14071 raft_consensus.cc:3037] T 89f4c1af687845b4a96759b172cd6aa6 P 367ccb48ccc64cda90859fb80a2e3b58 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:18.594127 14071 raft_consensus.cc:2454] T 89f4c1af687845b4a96759b172cd6aa6 P 367ccb48ccc64cda90859fb80a2e3b58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fad2ed6f21704cafbabf09532815df47 in term 1.
I20260620 19:10:18.594417 14273 leader_election.cc:302] T 89f4c1af687845b4a96759b172cd6aa6 P fad2ed6f21704cafbabf09532815df47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 367ccb48ccc64cda90859fb80a2e3b58, fad2ed6f21704cafbabf09532815df47; no voters:
I20260620 19:10:18.594395 14204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89f4c1af687845b4a96759b172cd6aa6" candidate_uuid: "fad2ed6f21704cafbabf09532815df47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e56807c1c3f542e7b8a10080b1de51b9"
I20260620 19:10:18.594525 14204 raft_consensus.cc:3037] T 89f4c1af687845b4a96759b172cd6aa6 P e56807c1c3f542e7b8a10080b1de51b9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:18.594647 14408 raft_consensus.cc:2781] T 89f4c1af687845b4a96759b172cd6aa6 P fad2ed6f21704cafbabf09532815df47 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:10:18.594893 14408 raft_consensus.cc:686] T 89f4c1af687845b4a96759b172cd6aa6 P fad2ed6f21704cafbabf09532815df47 [term 1 LEADER]: Becoming Leader. State: Replica: fad2ed6f21704cafbabf09532815df47, State: Running, Role: LEADER
I20260620 19:10:18.595245 14408 consensus_queue.cc:237] T 89f4c1af687845b4a96759b172cd6aa6 P fad2ed6f21704cafbabf09532815df47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56807c1c3f542e7b8a10080b1de51b9" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 37245 } } peers { permanent_uuid: "367ccb48ccc64cda90859fb80a2e3b58" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 36057 } } peers { permanent_uuid: "fad2ed6f21704cafbabf09532815df47" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 35437 } }
I20260620 19:10:18.595659 14204 raft_consensus.cc:2454] T 89f4c1af687845b4a96759b172cd6aa6 P e56807c1c3f542e7b8a10080b1de51b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fad2ed6f21704cafbabf09532815df47 in term 1.
I20260620 19:10:18.598796 13922 catalog_manager.cc:5495] T 89f4c1af687845b4a96759b172cd6aa6 P fad2ed6f21704cafbabf09532815df47 reported cstate change: term changed from 0 to 1, leader changed from <none> to fad2ed6f21704cafbabf09532815df47 (127.9.155.67). New cstate: current_term: 1 leader_uuid: "fad2ed6f21704cafbabf09532815df47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56807c1c3f542e7b8a10080b1de51b9" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 37245 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "367ccb48ccc64cda90859fb80a2e3b58" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 36057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fad2ed6f21704cafbabf09532815df47" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 35437 } health_report { overall_health: HEALTHY } } }
W20260620 19:10:18.610146 14118 tablet.cc:2369] T 89f4c1af687845b4a96759b172cd6aa6 P 367ccb48ccc64cda90859fb80a2e3b58: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:10:18.621570 14399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:18.617887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39300 (local address 127.9.155.67:35437)
0620 19:10:18.618025 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:18.618029 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:18.618059 (+ 30us) server_negotiation.cc:407] Connection header received
0620 19:10:18.618139 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:18.618141 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:18.618199 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:18.618341 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:10:18.618604 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:18.619134 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:18.620034 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:18.620283 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:18.620372 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:18.620698 (+ 326us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:10:18.621158 (+ 460us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:10:18.621256 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:18.621355 (+ 99us) server_negotiation.cc:299] Negotiation successful
0620 19:10:18.621412 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 19:10:18.627779 14204 raft_consensus.cc:1261] T 89f4c1af687845b4a96759b172cd6aa6 P e56807c1c3f542e7b8a10080b1de51b9 [term 1 FOLLOWER]: Refusing update from remote peer fad2ed6f21704cafbabf09532815df47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:10:18.627755 14071 raft_consensus.cc:1261] T 89f4c1af687845b4a96759b172cd6aa6 P 367ccb48ccc64cda90859fb80a2e3b58 [term 1 FOLLOWER]: Refusing update from remote peer fad2ed6f21704cafbabf09532815df47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:10:18.628687 14410 consensus_queue.cc:1035] T 89f4c1af687845b4a96759b172cd6aa6 P fad2ed6f21704cafbabf09532815df47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e56807c1c3f542e7b8a10080b1de51b9" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 37245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:10:18.628942 14408 consensus_queue.cc:1035] T 89f4c1af687845b4a96759b172cd6aa6 P fad2ed6f21704cafbabf09532815df47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "367ccb48ccc64cda90859fb80a2e3b58" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 36057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:10:18.637655 14414 mvcc.cc:205] Tried to move back new op lower bound from 7299000805890895872 to 7299000805769039872. Current Snapshot: MvccSnapshot[applied={T|T < 7299000805890895872}]
I20260620 19:10:18.637547 14415 mvcc.cc:205] Tried to move back new op lower bound from 7299000805890895872 to 7299000805769039872. Current Snapshot: MvccSnapshot[applied={T|T < 7299000805890895872}]
I20260620 19:10:18.646674 13926 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58580:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:10:18.646853 13926 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58580:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:10:18.649322 13926 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 815622646e3045c889d0f94cd715cade: Sending DeleteTablet for 3 replicas of tablet 89f4c1af687845b4a96759b172cd6aa6
I20260620 19:10:18.650137 14184 tablet_service.cc:1514] Processing DeleteTablet for tablet 89f4c1af687845b4a96759b172cd6aa6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:10:18 UTC) from {username='kudu', principal='kudu/127.9.155.126@KRBTEST.COM'} at 127.0.0.1:45944
I20260620 19:10:18.650213 14051 tablet_service.cc:1514] Processing DeleteTablet for tablet 89f4c1af687845b4a96759b172cd6aa6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:10:18 UTC) from {username='kudu', principal='kudu/127.9.155.126@KRBTEST.COM'} at 127.0.0.1:54138
I20260620 19:10:18.650703 14424 tablet_replica.cc:330] T 89f4c1af687845b4a96759b172cd6aa6 P 367ccb48ccc64cda90859fb80a2e3b58: stopping tablet replica
I20260620 19:10:18.651050 14423 tablet_replica.cc:330] T 89f4c1af687845b4a96759b172cd6aa6 P e56807c1c3f542e7b8a10080b1de51b9: stopping tablet replica
I20260620 19:10:18.651034 14317 tablet_service.cc:1514] Processing DeleteTablet for tablet 89f4c1af687845b4a96759b172cd6aa6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:10:18 UTC) from {username='kudu', principal='kudu/127.9.155.126@KRBTEST.COM'} at 127.0.0.1:39294
I20260620 19:10:18.651400 14423 raft_consensus.cc:2229] T 89f4c1af687845b4a96759b172cd6aa6 P e56807c1c3f542e7b8a10080b1de51b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:10:18.651544 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 13989
I20260620 19:10:18.651762 14423 raft_consensus.cc:2258] T 89f4c1af687845b4a96759b172cd6aa6 P e56807c1c3f542e7b8a10080b1de51b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:10:18.652526 14425 tablet_replica.cc:330] T 89f4c1af687845b4a96759b172cd6aa6 P fad2ed6f21704cafbabf09532815df47: stopping tablet replica
I20260620 19:10:18.652846 14425 raft_consensus.cc:2229] T 89f4c1af687845b4a96759b172cd6aa6 P fad2ed6f21704cafbabf09532815df47 [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:10:18.653247 14425 raft_consensus.cc:2258] T 89f4c1af687845b4a96759b172cd6aa6 P fad2ed6f21704cafbabf09532815df47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:10:18.653450 14423 ts_tablet_manager.cc:1905] T 89f4c1af687845b4a96759b172cd6aa6 P e56807c1c3f542e7b8a10080b1de51b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:10:18.654744 14425 ts_tablet_manager.cc:1905] T 89f4c1af687845b4a96759b172cd6aa6 P fad2ed6f21704cafbabf09532815df47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:10:18.656283 14423 ts_tablet_manager.cc:1918] T 89f4c1af687845b4a96759b172cd6aa6 P e56807c1c3f542e7b8a10080b1de51b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:10:18.656386 14423 log.cc:1196] T 89f4c1af687845b4a96759b172cd6aa6 P e56807c1c3f542e7b8a10080b1de51b9: Deleting WAL directory at /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/minicluster-data/ts-1/wal/wals/89f4c1af687845b4a96759b172cd6aa6
I20260620 19:10:18.656679 14423 ts_tablet_manager.cc:1939] T 89f4c1af687845b4a96759b172cd6aa6 P e56807c1c3f542e7b8a10080b1de51b9: Deleting consensus metadata
I20260620 19:10:18.657743 14425 ts_tablet_manager.cc:1918] T 89f4c1af687845b4a96759b172cd6aa6 P fad2ed6f21704cafbabf09532815df47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:10:18.657855 14425 log.cc:1196] T 89f4c1af687845b4a96759b172cd6aa6 P fad2ed6f21704cafbabf09532815df47: Deleting WAL directory at /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982595231230-9837-0/minicluster-data/ts-2/wal/wals/89f4c1af687845b4a96759b172cd6aa6
I20260620 19:10:18.657963 13911 catalog_manager.cc:4841] TS e56807c1c3f542e7b8a10080b1de51b9 (127.9.155.66:37245): tablet 89f4c1af687845b4a96759b172cd6aa6 (table test-table [id=d1a1f7d79a7d4f499c18d690f9f496ef]) successfully deleted
I20260620 19:10:18.658154 14425 ts_tablet_manager.cc:1939] T 89f4c1af687845b4a96759b172cd6aa6 P fad2ed6f21704cafbabf09532815df47: Deleting consensus metadata
I20260620 19:10:18.659308 13913 catalog_manager.cc:4841] TS fad2ed6f21704cafbabf09532815df47 (127.9.155.67:35437): tablet 89f4c1af687845b4a96759b172cd6aa6 (table test-table [id=d1a1f7d79a7d4f499c18d690f9f496ef]) successfully deleted
W20260620 19:10:18.660270 13914 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.155.65:36057 (error 108)
I20260620 19:10:18.661041 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 14122
I20260620 19:10:18.661284 14395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:18.660991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.65:36057 (local address 127.0.0.1:54150)
0620 19:10:18.661125 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:18.661197 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.65:36057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 19:10:18.662369 13914 catalog_manager.cc:4584] TS 367ccb48ccc64cda90859fb80a2e3b58 (127.9.155.65:36057): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 89f4c1af687845b4a96759b172cd6aa6: Network error: Client connection negotiation failed: client connection to 127.9.155.65:36057: connect: Connection refused (error 111)
I20260620 19:10:18.666662 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 14255
I20260620 19:10:18.672272 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 13896
2026-06-20T19:10:18Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2320 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-20T19:10:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:10:18Z Disabled control of system clock
I20260620 19:10:18.700898 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982595231230-9837-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982595231230-9837-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982595231230-9837-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:34737
--webserver_interface=127.9.155.126
--webserver_port=0
--builtin_ntp_servers=127.9.155.84:36107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.155.126:34737
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 19:10:18.811107 14430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:18.811411 14430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:18.811450 14430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:18.815614 14430 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:10:18.815685 14430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:18.815701 14430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:18.815716 14430 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:10:18.815737 14430 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:10:18.820632 14430 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:36107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982595231230-9837-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982595231230-9837-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.155.126:34737
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:34737
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.155.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14430
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982595231230-9837-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:10:18.821682 14430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:18.823010 14430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:10:18.831729 14436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:18.831729 14438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:18.831816 14430 server_base.cc:1034] running on GCE node
W20260620 19:10:18.831729 14435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:18.832693 14430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:18.833825 14430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:18.834998 14430 hybrid_clock.cc:648] HybridClock initialized: now 1781982618834982 us; error 32 us; skew 500 ppm
I20260620 19:10:18.836836 14430 webserver.cc:460] Webserver started at http://127.9.155.126:39157/ using document root <none> and password file <none>
I20260620 19:10:18.837393 14430 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:18.837458 14430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:18.837651 14430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:10:18.839648 14430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982595231230-9837-0/minicluster-data/master-0/data/instance:
uuid: "a994d7bd080b4116924dc2c604203c38"
format_stamp: "Formatted at 2026-06-20 19:10:18 on dist-test-slave-n3fx"
server_key: "af840a2f919c9b334bb5e093ee368835"
server_key_iv: "965bb6b48943f10a60ec8170c1a56af4"
server_key_version: "encryptionkey@0"
I20260620 19:10:18.840173 14430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982595231230-9837-0/minicluster-data/master-0/wal/instance:
uuid: "a994d7bd080b4116924dc2c604203c38"
format_stamp: "Formatted at 2026-06-20 19:10:18 on dist-test-slave-n3fx"
server_key: "af840a2f919c9b334bb5e093ee368835"
server_key_iv: "965bb6b48943f10a60ec8170c1a56af4"
server_key_version: "encryptionkey@0"
I20260620 19:10:18.843725 14430 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 19:10:18.846235 14444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:18.847579 14430 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 19:10:18.847723 14430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982595231230-9837-0/minicluster-data/master-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982595231230-9837-0/minicluster-data/master-0/wal
uuid: "a994d7bd080b4116924dc2c604203c38"
format_stamp: "Formatted at 2026-06-20 19:10:18 on dist-test-slave-n3fx"
server_key: "af840a2f919c9b334bb5e093ee368835"
server_key_iv: "965bb6b48943f10a60ec8170c1a56af4"
server_key_version: "encryptionkey@0"
I20260620 19:10:18.847829 14430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982595231230-9837-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982595231230-9837-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982595231230-9837-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:18.852726 14430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:18.853391 14430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:18.853556 14430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:18.860396 14430 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.126:34737
I20260620 19:10:18.860414 14496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.126:34737 every 8 connection(s)
I20260620 19:10:18.861358 14430 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
I20260620 19:10:18.864643 14497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:18.866318 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 14430
I20260620 19:10:18.866438 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982595231230-9837-0/minicluster-data/master-0/wal/instance
I20260620 19:10:18.866721 9837 external_mini_cluster.cc:1428] Setting key 85ae2005bbb6b119619fcab9c41ca21f
I20260620 19:10:18.871861 14497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a994d7bd080b4116924dc2c604203c38: Bootstrap starting.
I20260620 19:10:18.872531 14500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:18.868176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43320 (local address 127.9.155.126:34737)
0620 19:10:18.868832 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:18.868849 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:18.868884 (+ 35us) server_negotiation.cc:407] Connection header received
0620 19:10:18.869589 (+ 705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:18.869608 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:18.869991 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:18.870287 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:18.871157 (+ 870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:18.871162 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:18.871175 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:10:18.871186 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:18.871244 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:10:18.871260 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:18.871382 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:18.871537 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 19:10:18.871764 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":106,"threads_started":1}
I20260620 19:10:18.874142 14497 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a994d7bd080b4116924dc2c604203c38: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:18.874949 14497 log.cc:824] T 00000000000000000000000000000000 P a994d7bd080b4116924dc2c604203c38: Log is configured to *not* fsync() on all Append() calls
I20260620 19:10:18.877199 14497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a994d7bd080b4116924dc2c604203c38: No bootstrap required, opened a new log
I20260620 19:10:18.880986 14497 raft_consensus.cc:348] T 00000000000000000000000000000000 P a994d7bd080b4116924dc2c604203c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a994d7bd080b4116924dc2c604203c38" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 34737 } }
I20260620 19:10:18.881273 14497 raft_consensus.cc:374] T 00000000000000000000000000000000 P a994d7bd080b4116924dc2c604203c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:18.881333 14497 raft_consensus.cc:729] T 00000000000000000000000000000000 P a994d7bd080b4116924dc2c604203c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a994d7bd080b4116924dc2c604203c38, State: Initialized, Role: FOLLOWER
I20260620 19:10:18.881762 14497 consensus_queue.cc:260] T 00000000000000000000000000000000 P a994d7bd080b4116924dc2c604203c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a994d7bd080b4116924dc2c604203c38" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 34737 } }
I20260620 19:10:18.881906 14497 raft_consensus.cc:388] T 00000000000000000000000000000000 P a994d7bd080b4116924dc2c604203c38 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:10:18.881969 14497 raft_consensus.cc:482] T 00000000000000000000000000000000 P a994d7bd080b4116924dc2c604203c38 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:10:18.882061 14497 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a994d7bd080b4116924dc2c604203c38 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:18.883021 14497 raft_consensus.cc:504] T 00000000000000000000000000000000 P a994d7bd080b4116924dc2c604203c38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a994d7bd080b4116924dc2c604203c38" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 34737 } }
I20260620 19:10:18.883375 14497 leader_election.cc:302] T 00000000000000000000000000000000 P a994d7bd080b4116924dc2c604203c38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a994d7bd080b4116924dc2c604203c38; no voters:
I20260620 19:10:18.883677 14497 leader_election.cc:288] T 00000000000000000000000000000000 P a994d7bd080b4116924dc2c604203c38 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:10:18.883826 14502 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a994d7bd080b4116924dc2c604203c38 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:10:18.884060 14502 raft_consensus.cc:686] T 00000000000000000000000000000000 P a994d7bd080b4116924dc2c604203c38 [term 1 LEADER]: Becoming Leader. State: Replica: a994d7bd080b4116924dc2c604203c38, State: Running, Role: LEADER
I20260620 19:10:18.884413 14502 consensus_queue.cc:237] T 00000000000000000000000000000000 P a994d7bd080b4116924dc2c604203c38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a994d7bd080b4116924dc2c604203c38" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 34737 } }
I20260620 19:10:18.884953 14497 sys_catalog.cc:564] T 00000000000000000000000000000000 P a994d7bd080b4116924dc2c604203c38 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:10:18.886075 14503 sys_catalog.cc:455] T 00000000000000000000000000000000 P a994d7bd080b4116924dc2c604203c38 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a994d7bd080b4116924dc2c604203c38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a994d7bd080b4116924dc2c604203c38" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 34737 } } }
I20260620 19:10:18.886209 14503 sys_catalog.cc:458] T 00000000000000000000000000000000 P a994d7bd080b4116924dc2c604203c38 [sys.catalog]: This master's current role is: LEADER
I20260620 19:10:18.886607 14511 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:10:18.886648 14504 sys_catalog.cc:455] T 00000000000000000000000000000000 P a994d7bd080b4116924dc2c604203c38 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a994d7bd080b4116924dc2c604203c38. Latest consensus state: current_term: 1 leader_uuid: "a994d7bd080b4116924dc2c604203c38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a994d7bd080b4116924dc2c604203c38" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 34737 } } }
I20260620 19:10:18.887360 14504 sys_catalog.cc:458] T 00000000000000000000000000000000 P a994d7bd080b4116924dc2c604203c38 [sys.catalog]: This master's current role is: LEADER
I20260620 19:10:18.890066 14511 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:10:18.896183 14511 catalog_manager.cc:1347] Generated new cluster ID: ff1f58791731478c8e18195f7447a9ef
I20260620 19:10:18.896302 14511 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:10:18.906749 14511 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:10:18.907166 14511 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:10:18.913226 14511 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a994d7bd080b4116924dc2c604203c38: Generated new TSK 0
I20260620 19:10:18.914104 14511 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:10:18.921221 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982595231230-9837-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982595231230-9837-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982595231230-9837-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.65:0
--local_ip_for_outbound_sockets=127.9.155.65
--webserver_interface=127.9.155.65
--webserver_port=0
--tserver_master_addrs=127.9.155.126:34737
--builtin_ntp_servers=127.9.155.84:36107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 19:10:19.034099 14521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:19.034379 14521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:19.034435 14521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:19.038551 14521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:19.038614 14521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:19.038695 14521 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.65
I20260620 19:10:19.043632 14521 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:36107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982595231230-9837-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982595231230-9837-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.155.65
--webserver_port=0
--tserver_master_addrs=127.9.155.126:34737
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14521
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.65
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982595231230-9837-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:10:19.044749 14521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:19.046083 14521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:10:19.055656 14529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:19.055598 14526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:19.055625 14527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:19.056030 14521 server_base.cc:1034] running on GCE node
I20260620 19:10:19.056509 14521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:19.057191 14521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:19.058364 14521 hybrid_clock.cc:648] HybridClock initialized: now 1781982619058346 us; error 34 us; skew 500 ppm
I20260620 19:10:19.063578 14521 webserver.cc:460] Webserver started at http://127.9.155.65:36463/ using document root <none> and password file <none>
I20260620 19:10:19.064179 14521 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:19.064280 14521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:19.064491 14521 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:10:19.066656 14521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982595231230-9837-0/minicluster-data/ts-0/data/instance:
uuid: "83fa7c8f15414d5a82d57dcd80b2223e"
format_stamp: "Formatted at 2026-06-20 19:10:19 on dist-test-slave-n3fx"
server_key: "636f4f31dfaba2e716a57d841b60a507"
server_key_iv: "aea15cf365f98569d811ee16a6fa256a"
server_key_version: "encryptionkey@0"
I20260620 19:10:19.067230 14521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982595231230-9837-0/minicluster-data/ts-0/wal/instance:
uuid: "83fa7c8f15414d5a82d57dcd80b2223e"
format_stamp: "Formatted at 2026-06-20 19:10:19 on dist-test-slave-n3fx"
server_key: "636f4f31dfaba2e716a57d841b60a507"
server_key_iv: "aea15cf365f98569d811ee16a6fa256a"
server_key_version: "encryptionkey@0"
I20260620 19:10:19.071228 14521 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 19:10:19.073938 14535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:19.075278 14521 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 19:10:19.075413 14521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982595231230-9837-0/minicluster-data/ts-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982595231230-9837-0/minicluster-data/ts-0/wal
uuid: "83fa7c8f15414d5a82d57dcd80b2223e"
format_stamp: "Formatted at 2026-06-20 19:10:19 on dist-test-slave-n3fx"
server_key: "636f4f31dfaba2e716a57d841b60a507"
server_key_iv: "aea15cf365f98569d811ee16a6fa256a"
server_key_version: "encryptionkey@0"
I20260620 19:10:19.075537 14521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982595231230-9837-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982595231230-9837-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982595231230-9837-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:19.080303 14521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:19.080957 14521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:19.081131 14521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:19.081697 14521 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:10:19.082644 14521 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:10:19.082693 14521 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:19.082746 14521 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:10:19.082774 14521 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:19.091177 14521 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.65:46081
I20260620 19:10:19.091198 14648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.65:46081 every 8 connection(s)
I20260620 19:10:19.092092 14521 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
I20260620 19:10:19.097576 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 14521
I20260620 19:10:19.097684 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982595231230-9837-0/minicluster-data/ts-0/wal/instance
I20260620 19:10:19.097946 9837 external_mini_cluster.cc:1428] Setting key 4945651bf58188cd3c8f57ae314a8f2d
I20260620 19:10:19.098151 14500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:19.093934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.65:44503 (local address 127.9.155.126:34737)
0620 19:10:19.094074 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:19.094079 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:19.095139 (+ 1060us) server_negotiation.cc:407] Connection header received
0620 19:10:19.096475 (+ 1336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:19.096478 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:19.096515 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:19.096590 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:19.097649 (+ 1059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:19.097652 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:19.097655 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:10:19.097670 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:19.097692 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:10:19.097701 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:19.097784 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:19.097970 (+ 186us) server_negotiation.cc:299] Negotiation successful
0620 19:10:19.098021 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 19:10:19.099092 14651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:19.094334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:34737 (local address 127.9.155.65:44503)
0620 19:10:19.094995 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:19.095037 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:19.096256 (+ 1219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:19.096742 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:19.096748 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:19.096836 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:19.097381 (+ 545us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:10:19.097382 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:19.097423 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:10:19.097492 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:19.097821 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:19.097828 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:19.097833 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 19:10:19.097970 (+ 137us) client_negotiation.cc:240] Negotiation successful
0620 19:10:19.098193 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":107,"threads_started":1}
I20260620 19:10:19.099903 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982595231230-9837-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982595231230-9837-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982595231230-9837-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.66:0
--local_ip_for_outbound_sockets=127.9.155.66
--webserver_interface=127.9.155.66
--webserver_port=0
--tserver_master_addrs=127.9.155.126:34737
--builtin_ntp_servers=127.9.155.84:36107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 19:10:19.100277 14649 heartbeater.cc:347] Connected to a master server at 127.9.155.126:34737
I20260620 19:10:19.100513 14649 heartbeater.cc:464] Registering TS with master...
I20260620 19:10:19.100939 14649 heartbeater.cc:511] Master 127.9.155.126:34737 requested a full tablet report, sending...
I20260620 19:10:19.102432 14461 ts_manager.cc:194] Registered new tserver with Master: 83fa7c8f15414d5a82d57dcd80b2223e (127.9.155.65:46081)
W20260620 19:10:19.209947 14652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:19.210232 14652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:19.210275 14652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:19.214473 14652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:19.214551 14652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:19.214617 14652 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.66
I20260620 19:10:19.219684 14652 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:36107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982595231230-9837-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982595231230-9837-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.155.66
--webserver_port=0
--tserver_master_addrs=127.9.155.126:34737
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14652
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.66
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982595231230-9837-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:10:19.220862 14652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:19.222224 14652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:10:19.231560 14658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:19.231560 14660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:19.231580 14657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:19.231895 14652 server_base.cc:1034] running on GCE node
I20260620 19:10:19.232282 14652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:19.232914 14652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:19.234094 14652 hybrid_clock.cc:648] HybridClock initialized: now 1781982619234072 us; error 38 us; skew 500 ppm
I20260620 19:10:19.236033 14652 webserver.cc:460] Webserver started at http://127.9.155.66:33909/ using document root <none> and password file <none>
I20260620 19:10:19.236630 14652 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:19.236696 14652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:19.236889 14652 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:10:19.238902 14652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982595231230-9837-0/minicluster-data/ts-1/data/instance:
uuid: "cf2d2bfcf14740cd80381d27f8d29354"
format_stamp: "Formatted at 2026-06-20 19:10:19 on dist-test-slave-n3fx"
server_key: "beaae459d807a54abd1168a9f5aea918"
server_key_iv: "e3bd3d21a8650a1a6dfb46da4c0a0d7c"
server_key_version: "encryptionkey@0"
I20260620 19:10:19.239442 14652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982595231230-9837-0/minicluster-data/ts-1/wal/instance:
uuid: "cf2d2bfcf14740cd80381d27f8d29354"
format_stamp: "Formatted at 2026-06-20 19:10:19 on dist-test-slave-n3fx"
server_key: "beaae459d807a54abd1168a9f5aea918"
server_key_iv: "e3bd3d21a8650a1a6dfb46da4c0a0d7c"
server_key_version: "encryptionkey@0"
I20260620 19:10:19.243099 14652 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260620 19:10:19.245573 14666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:19.246965 14652 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 19:10:19.247099 14652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982595231230-9837-0/minicluster-data/ts-1/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982595231230-9837-0/minicluster-data/ts-1/wal
uuid: "cf2d2bfcf14740cd80381d27f8d29354"
format_stamp: "Formatted at 2026-06-20 19:10:19 on dist-test-slave-n3fx"
server_key: "beaae459d807a54abd1168a9f5aea918"
server_key_iv: "e3bd3d21a8650a1a6dfb46da4c0a0d7c"
server_key_version: "encryptionkey@0"
I20260620 19:10:19.247206 14652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982595231230-9837-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982595231230-9837-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982595231230-9837-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:19.252105 14652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:19.252760 14652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:19.252920 14652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:19.253463 14652 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:10:19.254369 14652 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:10:19.254415 14652 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:19.254464 14652 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:10:19.254493 14652 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:19.262583 14652 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.66:40841
I20260620 19:10:19.262612 14779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.66:40841 every 8 connection(s)
I20260620 19:10:19.263458 14652 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
I20260620 19:10:19.265861 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 14652
I20260620 19:10:19.265974 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982595231230-9837-0/minicluster-data/ts-1/wal/instance
I20260620 19:10:19.266291 9837 external_mini_cluster.cc:1428] Setting key 9480ce73f22d8f60973b4283df848332
I20260620 19:10:19.268539 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982595231230-9837-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982595231230-9837-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982595231230-9837-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.67:0
--local_ip_for_outbound_sockets=127.9.155.67
--webserver_interface=127.9.155.67
--webserver_port=0
--tserver_master_addrs=127.9.155.126:34737
--builtin_ntp_servers=127.9.155.84:36107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 19:10:19.270170 14500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:19.265282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.66:55279 (local address 127.9.155.126:34737)
0620 19:10:19.265440 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:19.265444 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:19.266695 (+ 1251us) server_negotiation.cc:407] Connection header received
0620 19:10:19.268189 (+ 1494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:19.268192 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:19.268263 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:19.268341 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:19.269615 (+ 1274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:19.269619 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:19.269621 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:10:19.269633 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:19.269654 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:10:19.269662 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:19.269766 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:19.269982 (+ 216us) server_negotiation.cc:299] Negotiation successful
0620 19:10:19.270031 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 19:10:19.271155 14782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:19.265708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:34737 (local address 127.9.155.66:55279)
0620 19:10:19.266520 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:19.266576 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:19.267927 (+ 1351us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:19.268561 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:19.268570 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:19.268696 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:19.269278 (+ 582us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:10:19.269279 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:19.269333 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:10:19.269431 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:19.269775 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:19.269785 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:19.269792 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 19:10:19.270011 (+ 219us) client_negotiation.cc:240] Negotiation successful
0620 19:10:19.270238 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":112,"threads_started":1}
I20260620 19:10:19.272421 14780 heartbeater.cc:347] Connected to a master server at 127.9.155.126:34737
I20260620 19:10:19.272646 14780 heartbeater.cc:464] Registering TS with master...
I20260620 19:10:19.273053 14780 heartbeater.cc:511] Master 127.9.155.126:34737 requested a full tablet report, sending...
I20260620 19:10:19.274075 14461 ts_manager.cc:194] Registered new tserver with Master: cf2d2bfcf14740cd80381d27f8d29354 (127.9.155.66:40841)
W20260620 19:10:19.380769 14783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:19.381067 14783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:19.381124 14783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:19.385308 14783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:19.385368 14783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:19.385452 14783 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.67
I20260620 19:10:19.390552 14783 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:36107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982595231230-9837-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982595231230-9837-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.155.67
--webserver_port=0
--tserver_master_addrs=127.9.155.126:34737
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14783
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.67
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982595231230-9837-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:10:19.391655 14783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:19.393070 14783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:10:19.402511 14788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:19.402511 14789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:19.402547 14791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:19.402907 14783 server_base.cc:1034] running on GCE node
I20260620 19:10:19.403280 14783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:19.403925 14783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:19.405102 14783 hybrid_clock.cc:648] HybridClock initialized: now 1781982619405057 us; error 60 us; skew 500 ppm
I20260620 19:10:19.406953 14783 webserver.cc:460] Webserver started at http://127.9.155.67:43617/ using document root <none> and password file <none>
I20260620 19:10:19.407480 14783 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:19.407537 14783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:19.407699 14783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:10:19.409723 14783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982595231230-9837-0/minicluster-data/ts-2/data/instance:
uuid: "7440814575a34d47aeebaac2ce46529f"
format_stamp: "Formatted at 2026-06-20 19:10:19 on dist-test-slave-n3fx"
server_key: "9dce44641545c027bcbe96b2b9d7cf4b"
server_key_iv: "c398be7eca65122b1c4486ae6943de87"
server_key_version: "encryptionkey@0"
I20260620 19:10:19.410223 14783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982595231230-9837-0/minicluster-data/ts-2/wal/instance:
uuid: "7440814575a34d47aeebaac2ce46529f"
format_stamp: "Formatted at 2026-06-20 19:10:19 on dist-test-slave-n3fx"
server_key: "9dce44641545c027bcbe96b2b9d7cf4b"
server_key_iv: "c398be7eca65122b1c4486ae6943de87"
server_key_version: "encryptionkey@0"
I20260620 19:10:19.413905 14783 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 19:10:19.416236 14797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:19.417537 14783 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:10:19.417650 14783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982595231230-9837-0/minicluster-data/ts-2/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982595231230-9837-0/minicluster-data/ts-2/wal
uuid: "7440814575a34d47aeebaac2ce46529f"
format_stamp: "Formatted at 2026-06-20 19:10:19 on dist-test-slave-n3fx"
server_key: "9dce44641545c027bcbe96b2b9d7cf4b"
server_key_iv: "c398be7eca65122b1c4486ae6943de87"
server_key_version: "encryptionkey@0"
I20260620 19:10:19.417752 14783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982595231230-9837-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982595231230-9837-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982595231230-9837-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:19.422293 14783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:19.422870 14783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:19.423015 14783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:19.423532 14783 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:10:19.424489 14783 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:10:19.424536 14783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:19.424587 14783 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:10:19.424612 14783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:19.432875 14783 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.67:45265
I20260620 19:10:19.432901 14910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.67:45265 every 8 connection(s)
I20260620 19:10:19.433725 14783 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
I20260620 19:10:19.435149 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 14783
I20260620 19:10:19.435240 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982595231230-9837-0/minicluster-data/ts-2/wal/instance
I20260620 19:10:19.435487 9837 external_mini_cluster.cc:1428] Setting key b7e46e4e3f6fea0d9694bc9893fde561
I20260620 19:10:19.440030 14500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:19.435580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.67:60457 (local address 127.9.155.126:34737)
0620 19:10:19.435715 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:19.435720 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:19.436994 (+ 1274us) server_negotiation.cc:407] Connection header received
0620 19:10:19.438306 (+ 1312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:19.438308 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:19.438351 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:19.438426 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:19.439597 (+ 1171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:19.439600 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:19.439602 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:10:19.439614 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:19.439633 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:10:19.439641 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:19.439731 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:19.439879 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 19:10:19.439922 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 19:10:19.440919 14913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:19.436000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:34737 (local address 127.9.155.67:60457)
0620 19:10:19.436844 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:19.436902 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:19.438125 (+ 1223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:19.438614 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:19.438621 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:19.438709 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:19.439259 (+ 550us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:10:19.439261 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:19.439302 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:10:19.439372 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:19.439726 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:19.439732 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:19.439735 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:10:19.439872 (+ 137us) client_negotiation.cc:240] Negotiation successful
0620 19:10:19.440072 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":155,"threads_started":1}
I20260620 19:10:19.442101 14911 heartbeater.cc:347] Connected to a master server at 127.9.155.126:34737
I20260620 19:10:19.442365 14911 heartbeater.cc:464] Registering TS with master...
I20260620 19:10:19.442761 14911 heartbeater.cc:511] Master 127.9.155.126:34737 requested a full tablet report, sending...
I20260620 19:10:19.443770 14461 ts_manager.cc:194] Registered new tserver with Master: 7440814575a34d47aeebaac2ce46529f (127.9.155.67:45265)
I20260620 19:10:19.449843 9837 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:10:19.453614 14500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:19.451812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43328 (local address 127.9.155.126:34737)
0620 19:10:19.451930 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:19.451935 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:19.452141 (+ 206us) server_negotiation.cc:407] Connection header received
0620 19:10:19.452397 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:19.452400 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:19.452440 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:19.452511 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:19.453154 (+ 643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:19.453157 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:19.453158 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:10:19.453170 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:19.453190 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:10:19.453198 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:19.453275 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:19.453468 (+ 193us) server_negotiation.cc:299] Negotiation successful
0620 19:10:19.453506 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 19:10:19.456426 14461 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43328:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:10:19.459316 14461 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:10:19.470803 14922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:19.467442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.67:45265 (local address 127.0.0.1:47696)
0620 19:10:19.467814 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:19.467866 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:19.467964 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:19.469186 (+ 1222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:19.469192 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:19.469265 (+ 73us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:19.469826 (+ 561us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:10:19.469828 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:19.469847 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:10:19.469918 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:19.470517 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:19.470520 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:19.470523 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:10:19.470597 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 19:10:19.470633 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":112,"threads_started":1}
I20260620 19:10:19.470969 14927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:19.468361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53630 (local address 127.9.155.65:46081)
0620 19:10:19.468740 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:19.468751 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:19.469034 (+ 283us) server_negotiation.cc:407] Connection header received
0620 19:10:19.469115 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:19.469120 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:19.469296 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:19.469426 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:19.470126 (+ 700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:19.470133 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:19.470146 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:10:19.470157 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:19.470200 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:10:19.470232 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:19.470336 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:19.470727 (+ 391us) server_negotiation.cc:299] Negotiation successful
0620 19:10:19.470799 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":102,"threads_started":1}
I20260620 19:10:19.470989 14924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:19.468179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.65:46081 (local address 127.0.0.1:53630)
0620 19:10:19.468936 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:19.468949 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:19.469028 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:19.469556 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:19.469559 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:19.469573 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:19.469967 (+ 394us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:10:19.469969 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:19.469981 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:10:19.470011 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:19.470385 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:19.470395 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:19.470401 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 19:10:19.470516 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 19:10:19.470585 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":693,"thread_start_us":118,"threads_started":1}
I20260620 19:10:19.470842 14923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:19.467624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47696 (local address 127.9.155.67:45265)
0620 19:10:19.468784 (+ 1160us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:19.468796 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:19.468815 (+ 19us) server_negotiation.cc:407] Connection header received
0620 19:10:19.468858 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:19.468862 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:19.469000 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:19.469172 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:19.470154 (+ 982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:19.470160 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:19.470289 (+ 129us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:10:19.470303 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:19.470349 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:10:19.470386 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:19.470531 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:19.470612 (+ 81us) server_negotiation.cc:299] Negotiation successful
0620 19:10:19.470683 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1039,"thread_start_us":156,"threads_started":1}
I20260620 19:10:19.471287 14926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:19.467681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.66:40841 (local address 127.0.0.1:37758)
0620 19:10:19.468568 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:19.468581 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:19.468660 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:19.470032 (+ 1372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:19.470035 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:19.470048 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:19.470421 (+ 373us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:10:19.470423 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:19.470433 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:10:19.470454 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:19.470802 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:19.470805 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:19.470807 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 19:10:19.470885 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 19:10:19.471176 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":833,"thread_start_us":112,"threads_started":1}
I20260620 19:10:19.471715 14925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:19.468152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37758 (local address 127.9.155.66:40841)
0620 19:10:19.468564 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:19.468576 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:19.468746 (+ 170us) server_negotiation.cc:407] Connection header received
0620 19:10:19.468806 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:19.468812 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:19.469000 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:19.469137 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:19.470578 (+ 1441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:19.470584 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:19.470593 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:10:19.470604 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:19.470642 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:10:19.470669 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:19.470763 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:19.471487 (+ 724us) server_negotiation.cc:299] Negotiation successful
0620 19:10:19.471561 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":112,"threads_started":1}
I20260620 19:10:19.473999 14845 tablet_service.cc:1467] Processing CreateTablet for tablet ee6d932ade3f4a9483f2b104300202c9 (DEFAULT_TABLE table=test-table [id=64deb134655b421898eeaa779ed11634]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:10:19.474179 14714 tablet_service.cc:1467] Processing CreateTablet for tablet ee6d932ade3f4a9483f2b104300202c9 (DEFAULT_TABLE table=test-table [id=64deb134655b421898eeaa779ed11634]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:10:19.474179 14583 tablet_service.cc:1467] Processing CreateTablet for tablet ee6d932ade3f4a9483f2b104300202c9 (DEFAULT_TABLE table=test-table [id=64deb134655b421898eeaa779ed11634]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:10:19.475018 14845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee6d932ade3f4a9483f2b104300202c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:19.475176 14583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee6d932ade3f4a9483f2b104300202c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:19.475175 14714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee6d932ade3f4a9483f2b104300202c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:19.482008 14928 tablet_bootstrap.cc:492] T ee6d932ade3f4a9483f2b104300202c9 P 7440814575a34d47aeebaac2ce46529f: Bootstrap starting.
I20260620 19:10:19.482524 14930 tablet_bootstrap.cc:492] T ee6d932ade3f4a9483f2b104300202c9 P 83fa7c8f15414d5a82d57dcd80b2223e: Bootstrap starting.
I20260620 19:10:19.482523 14929 tablet_bootstrap.cc:492] T ee6d932ade3f4a9483f2b104300202c9 P cf2d2bfcf14740cd80381d27f8d29354: Bootstrap starting.
I20260620 19:10:19.484001 14928 tablet_bootstrap.cc:654] T ee6d932ade3f4a9483f2b104300202c9 P 7440814575a34d47aeebaac2ce46529f: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:19.484606 14929 tablet_bootstrap.cc:654] T ee6d932ade3f4a9483f2b104300202c9 P cf2d2bfcf14740cd80381d27f8d29354: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:19.484606 14930 tablet_bootstrap.cc:654] T ee6d932ade3f4a9483f2b104300202c9 P 83fa7c8f15414d5a82d57dcd80b2223e: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:19.484808 14928 log.cc:824] T ee6d932ade3f4a9483f2b104300202c9 P 7440814575a34d47aeebaac2ce46529f: Log is configured to *not* fsync() on all Append() calls
I20260620 19:10:19.485416 14930 log.cc:824] T ee6d932ade3f4a9483f2b104300202c9 P 83fa7c8f15414d5a82d57dcd80b2223e: Log is configured to *not* fsync() on all Append() calls
I20260620 19:10:19.485416 14929 log.cc:824] T ee6d932ade3f4a9483f2b104300202c9 P cf2d2bfcf14740cd80381d27f8d29354: Log is configured to *not* fsync() on all Append() calls
I20260620 19:10:19.487001 14928 tablet_bootstrap.cc:492] T ee6d932ade3f4a9483f2b104300202c9 P 7440814575a34d47aeebaac2ce46529f: No bootstrap required, opened a new log
I20260620 19:10:19.487195 14928 ts_tablet_manager.cc:1397] T ee6d932ade3f4a9483f2b104300202c9 P 7440814575a34d47aeebaac2ce46529f: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260620 19:10:19.487298 14930 tablet_bootstrap.cc:492] T ee6d932ade3f4a9483f2b104300202c9 P 83fa7c8f15414d5a82d57dcd80b2223e: No bootstrap required, opened a new log
I20260620 19:10:19.487469 14930 ts_tablet_manager.cc:1397] T ee6d932ade3f4a9483f2b104300202c9 P 83fa7c8f15414d5a82d57dcd80b2223e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 19:10:19.487671 14929 tablet_bootstrap.cc:492] T ee6d932ade3f4a9483f2b104300202c9 P cf2d2bfcf14740cd80381d27f8d29354: No bootstrap required, opened a new log
I20260620 19:10:19.487856 14929 ts_tablet_manager.cc:1397] T ee6d932ade3f4a9483f2b104300202c9 P cf2d2bfcf14740cd80381d27f8d29354: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 19:10:19.490823 14930 raft_consensus.cc:348] T ee6d932ade3f4a9483f2b104300202c9 P 83fa7c8f15414d5a82d57dcd80b2223e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7440814575a34d47aeebaac2ce46529f" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 45265 } } peers { permanent_uuid: "83fa7c8f15414d5a82d57dcd80b2223e" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 46081 } } peers { permanent_uuid: "cf2d2bfcf14740cd80381d27f8d29354" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40841 } }
I20260620 19:10:19.491050 14930 raft_consensus.cc:374] T ee6d932ade3f4a9483f2b104300202c9 P 83fa7c8f15414d5a82d57dcd80b2223e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:19.491099 14930 raft_consensus.cc:729] T ee6d932ade3f4a9483f2b104300202c9 P 83fa7c8f15414d5a82d57dcd80b2223e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83fa7c8f15414d5a82d57dcd80b2223e, State: Initialized, Role: FOLLOWER
I20260620 19:10:19.491534 14930 consensus_queue.cc:260] T ee6d932ade3f4a9483f2b104300202c9 P 83fa7c8f15414d5a82d57dcd80b2223e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7440814575a34d47aeebaac2ce46529f" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 45265 } } peers { permanent_uuid: "83fa7c8f15414d5a82d57dcd80b2223e" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 46081 } } peers { permanent_uuid: "cf2d2bfcf14740cd80381d27f8d29354" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40841 } }
I20260620 19:10:19.491636 14928 raft_consensus.cc:348] T ee6d932ade3f4a9483f2b104300202c9 P 7440814575a34d47aeebaac2ce46529f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7440814575a34d47aeebaac2ce46529f" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 45265 } } peers { permanent_uuid: "83fa7c8f15414d5a82d57dcd80b2223e" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 46081 } } peers { permanent_uuid: "cf2d2bfcf14740cd80381d27f8d29354" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40841 } }
I20260620 19:10:19.491871 14928 raft_consensus.cc:374] T ee6d932ade3f4a9483f2b104300202c9 P 7440814575a34d47aeebaac2ce46529f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:19.491917 14928 raft_consensus.cc:729] T ee6d932ade3f4a9483f2b104300202c9 P 7440814575a34d47aeebaac2ce46529f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7440814575a34d47aeebaac2ce46529f, State: Initialized, Role: FOLLOWER
I20260620 19:10:19.492061 14929 raft_consensus.cc:348] T ee6d932ade3f4a9483f2b104300202c9 P cf2d2bfcf14740cd80381d27f8d29354 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7440814575a34d47aeebaac2ce46529f" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 45265 } } peers { permanent_uuid: "83fa7c8f15414d5a82d57dcd80b2223e" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 46081 } } peers { permanent_uuid: "cf2d2bfcf14740cd80381d27f8d29354" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40841 } }
I20260620 19:10:19.492384 14929 raft_consensus.cc:374] T ee6d932ade3f4a9483f2b104300202c9 P cf2d2bfcf14740cd80381d27f8d29354 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:19.492439 14929 raft_consensus.cc:729] T ee6d932ade3f4a9483f2b104300202c9 P cf2d2bfcf14740cd80381d27f8d29354 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf2d2bfcf14740cd80381d27f8d29354, State: Initialized, Role: FOLLOWER
I20260620 19:10:19.492455 14928 consensus_queue.cc:260] T ee6d932ade3f4a9483f2b104300202c9 P 7440814575a34d47aeebaac2ce46529f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7440814575a34d47aeebaac2ce46529f" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 45265 } } peers { permanent_uuid: "83fa7c8f15414d5a82d57dcd80b2223e" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 46081 } } peers { permanent_uuid: "cf2d2bfcf14740cd80381d27f8d29354" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40841 } }
I20260620 19:10:19.492589 14930 ts_tablet_manager.cc:1428] T ee6d932ade3f4a9483f2b104300202c9 P 83fa7c8f15414d5a82d57dcd80b2223e: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260620 19:10:19.492863 14929 consensus_queue.cc:260] T ee6d932ade3f4a9483f2b104300202c9 P cf2d2bfcf14740cd80381d27f8d29354 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7440814575a34d47aeebaac2ce46529f" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 45265 } } peers { permanent_uuid: "83fa7c8f15414d5a82d57dcd80b2223e" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 46081 } } peers { permanent_uuid: "cf2d2bfcf14740cd80381d27f8d29354" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40841 } }
I20260620 19:10:19.493016 14649 heartbeater.cc:503] Master 127.9.155.126:34737 was elected leader, sending a full tablet report...
I20260620 19:10:19.493372 14928 ts_tablet_manager.cc:1428] T ee6d932ade3f4a9483f2b104300202c9 P 7440814575a34d47aeebaac2ce46529f: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 19:10:19.493697 14929 ts_tablet_manager.cc:1428] T ee6d932ade3f4a9483f2b104300202c9 P cf2d2bfcf14740cd80381d27f8d29354: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 19:10:19.493690 14911 heartbeater.cc:503] Master 127.9.155.126:34737 was elected leader, sending a full tablet report...
I20260620 19:10:19.494031 14780 heartbeater.cc:503] Master 127.9.155.126:34737 was elected leader, sending a full tablet report...
I20260620 19:10:19.506639 14935 raft_consensus.cc:482] T ee6d932ade3f4a9483f2b104300202c9 P 7440814575a34d47aeebaac2ce46529f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:10:19.506917 14935 raft_consensus.cc:504] T ee6d932ade3f4a9483f2b104300202c9 P 7440814575a34d47aeebaac2ce46529f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7440814575a34d47aeebaac2ce46529f" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 45265 } } peers { permanent_uuid: "83fa7c8f15414d5a82d57dcd80b2223e" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 46081 } } peers { permanent_uuid: "cf2d2bfcf14740cd80381d27f8d29354" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40841 } }
I20260620 19:10:19.508064 14935 leader_election.cc:288] T ee6d932ade3f4a9483f2b104300202c9 P 7440814575a34d47aeebaac2ce46529f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83fa7c8f15414d5a82d57dcd80b2223e (127.9.155.65:46081), cf2d2bfcf14740cd80381d27f8d29354 (127.9.155.66:40841)
I20260620 19:10:19.509925 14913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:19.508361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.66:40841 (local address 127.9.155.67:59977)
0620 19:10:19.508565 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:19.508580 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:19.508671 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:19.508913 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:19.508916 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:19.508930 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:19.509362 (+ 432us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:10:19.509364 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:19.509375 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:10:19.509405 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:19.509684 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:19.509687 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:19.509691 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:10:19.509769 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 19:10:19.509806 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131}
I20260620 19:10:19.509941 14925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:19.508462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.67:59977 (local address 127.9.155.66:40841)
0620 19:10:19.508593 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:19.508597 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:19.508685 (+ 88us) server_negotiation.cc:407] Connection header received
0620 19:10:19.508785 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:19.508787 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:19.508822 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:19.508888 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:19.509553 (+ 665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:19.509557 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:19.509559 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:10:19.509572 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:19.509593 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:10:19.509604 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:19.509688 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:19.509780 (+ 92us) server_negotiation.cc:299] Negotiation successful
0620 19:10:19.509822 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 19:10:19.510340 14927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:19.508479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.67:41085 (local address 127.9.155.65:46081)
0620 19:10:19.508593 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:19.508598 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:19.508836 (+ 238us) server_negotiation.cc:407] Connection header received
0620 19:10:19.508923 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:19.508926 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:19.508959 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:19.509022 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:19.509664 (+ 642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:19.509667 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:19.509670 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:10:19.509683 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:19.509705 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:10:19.509715 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:19.509793 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:19.510184 (+ 391us) server_negotiation.cc:299] Negotiation successful
0620 19:10:19.510226 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 19:10:19.510656 14937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:19.508400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.65:46081 (local address 127.9.155.67:41085)
0620 19:10:19.508739 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:19.508750 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:19.508843 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:19.509047 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:19.509050 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:19.509064 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:19.509507 (+ 443us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:10:19.509509 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:19.509519 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:10:19.509545 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:19.509983 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:19.509987 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:19.509990 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:10:19.510475 (+ 485us) client_negotiation.cc:240] Negotiation successful
0620 19:10:19.510514 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":132,"threads_started":1}
I20260620 19:10:19.510584 14734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee6d932ade3f4a9483f2b104300202c9" candidate_uuid: "7440814575a34d47aeebaac2ce46529f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf2d2bfcf14740cd80381d27f8d29354" is_pre_election: true
I20260620 19:10:19.510902 14734 raft_consensus.cc:2454] T ee6d932ade3f4a9483f2b104300202c9 P cf2d2bfcf14740cd80381d27f8d29354 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7440814575a34d47aeebaac2ce46529f in term 0.
I20260620 19:10:19.511255 14603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee6d932ade3f4a9483f2b104300202c9" candidate_uuid: "7440814575a34d47aeebaac2ce46529f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83fa7c8f15414d5a82d57dcd80b2223e" is_pre_election: true
I20260620 19:10:19.511351 14799 leader_election.cc:302] T ee6d932ade3f4a9483f2b104300202c9 P 7440814575a34d47aeebaac2ce46529f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7440814575a34d47aeebaac2ce46529f, cf2d2bfcf14740cd80381d27f8d29354; no voters:
I20260620 19:10:19.511576 14603 raft_consensus.cc:2454] T ee6d932ade3f4a9483f2b104300202c9 P 83fa7c8f15414d5a82d57dcd80b2223e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7440814575a34d47aeebaac2ce46529f in term 0.
I20260620 19:10:19.511622 14935 raft_consensus.cc:2781] T ee6d932ade3f4a9483f2b104300202c9 P 7440814575a34d47aeebaac2ce46529f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:10:19.511694 14935 raft_consensus.cc:482] T ee6d932ade3f4a9483f2b104300202c9 P 7440814575a34d47aeebaac2ce46529f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:10:19.511741 14935 raft_consensus.cc:3037] T ee6d932ade3f4a9483f2b104300202c9 P 7440814575a34d47aeebaac2ce46529f [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:19.512804 14935 raft_consensus.cc:504] T ee6d932ade3f4a9483f2b104300202c9 P 7440814575a34d47aeebaac2ce46529f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7440814575a34d47aeebaac2ce46529f" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 45265 } } peers { permanent_uuid: "83fa7c8f15414d5a82d57dcd80b2223e" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 46081 } } peers { permanent_uuid: "cf2d2bfcf14740cd80381d27f8d29354" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40841 } }
I20260620 19:10:19.513165 14935 leader_election.cc:288] T ee6d932ade3f4a9483f2b104300202c9 P 7440814575a34d47aeebaac2ce46529f [CANDIDATE]: Term 1 election: Requested vote from peers 83fa7c8f15414d5a82d57dcd80b2223e (127.9.155.65:46081), cf2d2bfcf14740cd80381d27f8d29354 (127.9.155.66:40841)
I20260620 19:10:19.513481 14734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee6d932ade3f4a9483f2b104300202c9" candidate_uuid: "7440814575a34d47aeebaac2ce46529f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf2d2bfcf14740cd80381d27f8d29354"
I20260620 19:10:19.513501 14603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee6d932ade3f4a9483f2b104300202c9" candidate_uuid: "7440814575a34d47aeebaac2ce46529f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83fa7c8f15414d5a82d57dcd80b2223e"
I20260620 19:10:19.513599 14734 raft_consensus.cc:3037] T ee6d932ade3f4a9483f2b104300202c9 P cf2d2bfcf14740cd80381d27f8d29354 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:19.513614 14603 raft_consensus.cc:3037] T ee6d932ade3f4a9483f2b104300202c9 P 83fa7c8f15414d5a82d57dcd80b2223e [term 0 FOLLOWER]: Advancing to term 1
W20260620 19:10:19.514334 14781 tablet.cc:2369] T ee6d932ade3f4a9483f2b104300202c9 P cf2d2bfcf14740cd80381d27f8d29354: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:10:19.514760 14734 raft_consensus.cc:2454] T ee6d932ade3f4a9483f2b104300202c9 P cf2d2bfcf14740cd80381d27f8d29354 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7440814575a34d47aeebaac2ce46529f in term 1.
I20260620 19:10:19.514750 14603 raft_consensus.cc:2454] T ee6d932ade3f4a9483f2b104300202c9 P 83fa7c8f15414d5a82d57dcd80b2223e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7440814575a34d47aeebaac2ce46529f in term 1.
I20260620 19:10:19.515025 14799 leader_election.cc:302] T ee6d932ade3f4a9483f2b104300202c9 P 7440814575a34d47aeebaac2ce46529f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7440814575a34d47aeebaac2ce46529f, cf2d2bfcf14740cd80381d27f8d29354; no voters:
I20260620 19:10:19.515228 14935 raft_consensus.cc:2781] T ee6d932ade3f4a9483f2b104300202c9 P 7440814575a34d47aeebaac2ce46529f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:10:19.515435 14935 raft_consensus.cc:686] T ee6d932ade3f4a9483f2b104300202c9 P 7440814575a34d47aeebaac2ce46529f [term 1 LEADER]: Becoming Leader. State: Replica: 7440814575a34d47aeebaac2ce46529f, State: Running, Role: LEADER
I20260620 19:10:19.515724 14935 consensus_queue.cc:237] T ee6d932ade3f4a9483f2b104300202c9 P 7440814575a34d47aeebaac2ce46529f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7440814575a34d47aeebaac2ce46529f" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 45265 } } peers { permanent_uuid: "83fa7c8f15414d5a82d57dcd80b2223e" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 46081 } } peers { permanent_uuid: "cf2d2bfcf14740cd80381d27f8d29354" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40841 } }
I20260620 19:10:19.518847 14461 catalog_manager.cc:5495] T ee6d932ade3f4a9483f2b104300202c9 P 7440814575a34d47aeebaac2ce46529f reported cstate change: term changed from 0 to 1, leader changed from <none> to 7440814575a34d47aeebaac2ce46529f (127.9.155.67). New cstate: current_term: 1 leader_uuid: "7440814575a34d47aeebaac2ce46529f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7440814575a34d47aeebaac2ce46529f" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 45265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83fa7c8f15414d5a82d57dcd80b2223e" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 46081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf2d2bfcf14740cd80381d27f8d29354" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40841 } health_report { overall_health: UNKNOWN } } }
I20260620 19:10:19.539966 14923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:19.538558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47704 (local address 127.9.155.67:45265)
0620 19:10:19.538687 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:19.538692 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:19.538706 (+ 14us) server_negotiation.cc:407] Connection header received
0620 19:10:19.538789 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:19.538792 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:19.538850 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:19.538945 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:19.539596 (+ 651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:19.539600 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:19.539603 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:10:19.539615 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:19.539636 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:10:19.539644 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:19.539719 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:19.539783 (+ 64us) server_negotiation.cc:299] Negotiation successful
0620 19:10:19.539823 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 19:10:19.545866 14734 raft_consensus.cc:1261] T ee6d932ade3f4a9483f2b104300202c9 P cf2d2bfcf14740cd80381d27f8d29354 [term 1 FOLLOWER]: Refusing update from remote peer 7440814575a34d47aeebaac2ce46529f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:10:19.545890 14603 raft_consensus.cc:1261] T ee6d932ade3f4a9483f2b104300202c9 P 83fa7c8f15414d5a82d57dcd80b2223e [term 1 FOLLOWER]: Refusing update from remote peer 7440814575a34d47aeebaac2ce46529f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:10:19.546537 14935 consensus_queue.cc:1035] T ee6d932ade3f4a9483f2b104300202c9 P 7440814575a34d47aeebaac2ce46529f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf2d2bfcf14740cd80381d27f8d29354" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:10:19.546752 14938 consensus_queue.cc:1035] T ee6d932ade3f4a9483f2b104300202c9 P 7440814575a34d47aeebaac2ce46529f [LEADER]: Connected to new peer: Peer: permanent_uuid: "83fa7c8f15414d5a82d57dcd80b2223e" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 46081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:10:19.563334 14461 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43328:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:10:19.563506 14461 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43328:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:10:19.565986 14461 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a994d7bd080b4116924dc2c604203c38: Sending DeleteTablet for 3 replicas of tablet ee6d932ade3f4a9483f2b104300202c9
I20260620 19:10:19.566895 14714 tablet_service.cc:1514] Processing DeleteTablet for tablet ee6d932ade3f4a9483f2b104300202c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:10:19 UTC) from {username='slave'} at 127.0.0.1:37758
I20260620 19:10:19.567117 14845 tablet_service.cc:1514] Processing DeleteTablet for tablet ee6d932ade3f4a9483f2b104300202c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:10:19 UTC) from {username='slave'} at 127.0.0.1:47696
I20260620 19:10:19.567462 14583 tablet_service.cc:1514] Processing DeleteTablet for tablet ee6d932ade3f4a9483f2b104300202c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:10:19 UTC) from {username='slave'} at 127.0.0.1:53630
I20260620 19:10:19.567581 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 14521
I20260620 19:10:19.567848 14952 tablet_replica.cc:330] T ee6d932ade3f4a9483f2b104300202c9 P 7440814575a34d47aeebaac2ce46529f: stopping tablet replica
I20260620 19:10:19.568532 14952 raft_consensus.cc:2229] T ee6d932ade3f4a9483f2b104300202c9 P 7440814575a34d47aeebaac2ce46529f [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:10:19.568953 14952 raft_consensus.cc:2258] T ee6d932ade3f4a9483f2b104300202c9 P 7440814575a34d47aeebaac2ce46529f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:10:19.569320 14951 tablet_replica.cc:330] T ee6d932ade3f4a9483f2b104300202c9 P cf2d2bfcf14740cd80381d27f8d29354: stopping tablet replica
I20260620 19:10:19.569619 14951 raft_consensus.cc:2229] T ee6d932ade3f4a9483f2b104300202c9 P cf2d2bfcf14740cd80381d27f8d29354 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:10:19.569923 14951 raft_consensus.cc:2258] T ee6d932ade3f4a9483f2b104300202c9 P cf2d2bfcf14740cd80381d27f8d29354 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:10:19.570960 14952 ts_tablet_manager.cc:1905] T ee6d932ade3f4a9483f2b104300202c9 P 7440814575a34d47aeebaac2ce46529f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:10:19.573617 14951 ts_tablet_manager.cc:1905] T ee6d932ade3f4a9483f2b104300202c9 P cf2d2bfcf14740cd80381d27f8d29354: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:10:19.573889 14952 ts_tablet_manager.cc:1918] T ee6d932ade3f4a9483f2b104300202c9 P 7440814575a34d47aeebaac2ce46529f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:10:19.574044 14952 log.cc:1196] T ee6d932ade3f4a9483f2b104300202c9 P 7440814575a34d47aeebaac2ce46529f: Deleting WAL directory at /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982595231230-9837-0/minicluster-data/ts-2/wal/wals/ee6d932ade3f4a9483f2b104300202c9
I20260620 19:10:19.574404 14952 ts_tablet_manager.cc:1939] T ee6d932ade3f4a9483f2b104300202c9 P 7440814575a34d47aeebaac2ce46529f: Deleting consensus metadata
W20260620 19:10:19.575657 14448 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.155.65:46081 (error 108)
I20260620 19:10:19.576172 14448 catalog_manager.cc:4841] TS 7440814575a34d47aeebaac2ce46529f (127.9.155.67:45265): tablet ee6d932ade3f4a9483f2b104300202c9 (table test-table [id=64deb134655b421898eeaa779ed11634]) successfully deleted
I20260620 19:10:19.576401 14951 ts_tablet_manager.cc:1918] T ee6d932ade3f4a9483f2b104300202c9 P cf2d2bfcf14740cd80381d27f8d29354: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:10:19.576498 14951 log.cc:1196] T ee6d932ade3f4a9483f2b104300202c9 P cf2d2bfcf14740cd80381d27f8d29354: Deleting WAL directory at /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982595231230-9837-0/minicluster-data/ts-1/wal/wals/ee6d932ade3f4a9483f2b104300202c9
I20260620 19:10:19.576794 14951 ts_tablet_manager.cc:1939] T ee6d932ade3f4a9483f2b104300202c9 P cf2d2bfcf14740cd80381d27f8d29354: Deleting consensus metadata
W20260620 19:10:19.577160 14926 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:10:19.576531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.65:46081 (local address 127.0.0.1:53632)
0620 19:10:19.576712 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:19.576748 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:19.576838 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:19.577055 (+ 217us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.65:46081: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":100}
I20260620 19:10:19.577273 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 14652
W20260620 19:10:19.577419 14448 catalog_manager.cc:4584] TS 83fa7c8f15414d5a82d57dcd80b2223e (127.9.155.65:46081): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ee6d932ade3f4a9483f2b104300202c9: Network error: Client connection negotiation failed: client connection to 127.9.155.65:46081: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 19:10:19.582799 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 14783
I20260620 19:10:19.583087 14926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:19.582770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.66:40841 (local address 127.0.0.1:37768)
0620 19:10:19.582913 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:19.582990 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.66:40841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 19:10:19.588420 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 14430
2026-06-20T19:10:19Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (914 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260620 19:10:19.607949 9837 mini_oidc.cc:152] Starting JWKS server
I20260620 19:10:19.609511 9837 webserver.cc:460] Webserver started at https://127.0.0.1:40165/ using document root <none> and password file <none>
I20260620 19:10:19.609648 9837 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 19:10:19.609865 9837 webserver.cc:460] Webserver started at http://127.0.0.1:41011/ using document root <none> and password file <none>
2026-06-20T19:10:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:10:19Z Disabled control of system clock
I20260620 19:10:19.618346 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982595231230-9837-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982595231230-9837-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982595231230-9837-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:44923
--webserver_interface=127.9.155.126
--webserver_port=0
--builtin_ntp_servers=127.9.155.84:44689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.155.126:44923
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40165/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982595231230-9837-0/testchainca.cert with env {}
W20260620 19:10:19.729526 14959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:19.729846 14959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:19.729887 14959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:19.734225 14959 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:10:19.734308 14959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:19.734323 14959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:19.734339 14959 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:10:19.734354 14959 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 19:10:19.734369 14959 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 19:10:19.734381 14959 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40165/jwks.json
I20260620 19:10:19.740037 14959 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:44689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982595231230-9837-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982595231230-9837-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.155.126:44923
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:44923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40165/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.155.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982595231230-9837-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14959
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982595231230-9837-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:10:19.741147 14959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:19.742101 14959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 19:10:19.750923 14959 server_base.cc:1034] running on GCE node
W20260620 19:10:19.750970 14964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:19.750865 14967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:19.750880 14965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:19.751863 14959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:19.753086 14959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:19.754254 14959 hybrid_clock.cc:648] HybridClock initialized: now 1781982619754261 us; error 63 us; skew 500 ppm
I20260620 19:10:19.756197 14959 webserver.cc:460] Webserver started at http://127.9.155.126:32869/ using document root <none> and password file <none>
I20260620 19:10:19.756809 14959 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:19.756878 14959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:19.757067 14959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:10:19.759099 14959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982595231230-9837-0/minicluster-data/master-0/data/instance:
uuid: "8daf955d620d4f90942c66a45d69147a"
format_stamp: "Formatted at 2026-06-20 19:10:19 on dist-test-slave-n3fx"
server_key: "3e55f82fe5e7abb616639dd806e2e180"
server_key_iv: "7df4692570750f358bd90f9c3eb406c9"
server_key_version: "encryptionkey@0"
I20260620 19:10:19.759637 14959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982595231230-9837-0/minicluster-data/master-0/wal/instance:
uuid: "8daf955d620d4f90942c66a45d69147a"
format_stamp: "Formatted at 2026-06-20 19:10:19 on dist-test-slave-n3fx"
server_key: "3e55f82fe5e7abb616639dd806e2e180"
server_key_iv: "7df4692570750f358bd90f9c3eb406c9"
server_key_version: "encryptionkey@0"
I20260620 19:10:19.763433 14959 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260620 19:10:19.766064 14973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:19.767462 14959 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 19:10:19.767596 14959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982595231230-9837-0/minicluster-data/master-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982595231230-9837-0/minicluster-data/master-0/wal
uuid: "8daf955d620d4f90942c66a45d69147a"
format_stamp: "Formatted at 2026-06-20 19:10:19 on dist-test-slave-n3fx"
server_key: "3e55f82fe5e7abb616639dd806e2e180"
server_key_iv: "7df4692570750f358bd90f9c3eb406c9"
server_key_version: "encryptionkey@0"
I20260620 19:10:19.767721 14959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982595231230-9837-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982595231230-9837-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982595231230-9837-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:19.805332 14959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:19.806036 14959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:19.806208 14959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:19.812866 14959 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.126:44923
I20260620 19:10:19.812901 15025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.126:44923 every 8 connection(s)
I20260620 19:10:19.813814 14959 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
I20260620 19:10:19.814334 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 14959
I20260620 19:10:19.814505 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982595231230-9837-0/minicluster-data/master-0/wal/instance
I20260620 19:10:19.814798 9837 external_mini_cluster.cc:1428] Setting key 147fd205cfcd819c3c49b7f22cc8cbaa
I20260620 19:10:19.816890 15026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:19.822556 15026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8daf955d620d4f90942c66a45d69147a: Bootstrap starting.
I20260620 19:10:19.824261 15029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:19.816260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36192 (local address 127.9.155.126:44923)
0620 19:10:19.816903 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:19.816920 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:19.816956 (+ 36us) server_negotiation.cc:407] Connection header received
0620 19:10:19.817638 (+ 682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:19.817657 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:19.818068 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:19.818403 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:19.818833 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:19.819803 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:19.820729 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:19.821196 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:19.821340 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:19.822234 (+ 894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:19.822239 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:19.822252 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:10:19.822650 (+ 398us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:19.822717 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:10:19.822743 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:19.822842 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:19.823014 (+ 172us) server_negotiation.cc:299] Negotiation successful
0620 19:10:19.823296 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":119,"threads_started":1}
I20260620 19:10:19.825191 15026 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8daf955d620d4f90942c66a45d69147a: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:19.825969 15026 log.cc:824] T 00000000000000000000000000000000 P 8daf955d620d4f90942c66a45d69147a: Log is configured to *not* fsync() on all Append() calls
I20260620 19:10:19.827574 9837 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:10:19.828337 15026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8daf955d620d4f90942c66a45d69147a: No bootstrap required, opened a new log
I20260620 19:10:19.832129 15026 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8daf955d620d4f90942c66a45d69147a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8daf955d620d4f90942c66a45d69147a" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 44923 } }
I20260620 19:10:19.832448 15026 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8daf955d620d4f90942c66a45d69147a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:19.832515 15026 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8daf955d620d4f90942c66a45d69147a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8daf955d620d4f90942c66a45d69147a, State: Initialized, Role: FOLLOWER
I20260620 19:10:19.832926 15026 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8daf955d620d4f90942c66a45d69147a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8daf955d620d4f90942c66a45d69147a" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 44923 } }
I20260620 19:10:19.833045 15026 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8daf955d620d4f90942c66a45d69147a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:10:19.833084 15026 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8daf955d620d4f90942c66a45d69147a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:10:19.833153 15026 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8daf955d620d4f90942c66a45d69147a [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:19.834079 15026 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8daf955d620d4f90942c66a45d69147a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8daf955d620d4f90942c66a45d69147a" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 44923 } }
I20260620 19:10:19.834390 15026 leader_election.cc:302] T 00000000000000000000000000000000 P 8daf955d620d4f90942c66a45d69147a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8daf955d620d4f90942c66a45d69147a; no voters:
I20260620 19:10:19.834651 15026 leader_election.cc:288] T 00000000000000000000000000000000 P 8daf955d620d4f90942c66a45d69147a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:10:19.834817 15031 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8daf955d620d4f90942c66a45d69147a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:10:19.835027 15031 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8daf955d620d4f90942c66a45d69147a [term 1 LEADER]: Becoming Leader. State: Replica: 8daf955d620d4f90942c66a45d69147a, State: Running, Role: LEADER
I20260620 19:10:19.835345 15031 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8daf955d620d4f90942c66a45d69147a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8daf955d620d4f90942c66a45d69147a" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 44923 } }
I20260620 19:10:19.835906 15026 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8daf955d620d4f90942c66a45d69147a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:10:19.836943 15032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8daf955d620d4f90942c66a45d69147a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8daf955d620d4f90942c66a45d69147a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8daf955d620d4f90942c66a45d69147a" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 44923 } } }
I20260620 19:10:19.837078 15032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8daf955d620d4f90942c66a45d69147a [sys.catalog]: This master's current role is: LEADER
I20260620 19:10:19.837350 15033 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8daf955d620d4f90942c66a45d69147a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8daf955d620d4f90942c66a45d69147a. Latest consensus state: current_term: 1 leader_uuid: "8daf955d620d4f90942c66a45d69147a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8daf955d620d4f90942c66a45d69147a" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 44923 } } }
I20260620 19:10:19.837436 15033 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8daf955d620d4f90942c66a45d69147a [sys.catalog]: This master's current role is: LEADER
I20260620 19:10:19.837532 15040 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:10:19.840449 15040 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:10:19.846088 15040 catalog_manager.cc:1347] Generated new cluster ID: 58ff0c962d7d4823b00a7cbbaaed79a4
I20260620 19:10:19.846163 15040 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:10:19.864195 15040 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:10:19.865265 15040 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:10:19.870760 15040 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8daf955d620d4f90942c66a45d69147a: Generated new TSK 0
I20260620 19:10:19.871902 15040 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 19:10:19.905874 15056 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260620 19:10:19.905979 15056 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 19:10:19.906623 15029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:19.888539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36198 (local address 127.9.155.126:44923)
0620 19:10:19.888690 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:19.888694 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:19.888824 (+ 130us) server_negotiation.cc:407] Connection header received
0620 19:10:19.889028 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:19.889031 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:19.889098 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:19.889213 (+ 115us) server_negotiation.cc:226] Negotiated authn=JWT
0620 19:10:19.889445 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:19.890280 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:19.891157 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:19.891495 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:19.891588 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:19.891938 (+ 350us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 19:10:19.906177 (+ 14239us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40165/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 19:10:19.906474 (+ 297us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36198: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40165/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":72}
W20260620 19:10:19.906689 15029 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36198: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40165/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260620 19:10:19.907734 15055 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:10:19.888438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:44923 (local address 127.0.0.1:36198)
0620 19:10:19.888736 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:19.888749 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:19.888871 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:19.889231 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:19.889233 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:19.889239 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0620 19:10:19.889345 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:19.889350 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:19.890421 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:19.890425 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:19.891052 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:19.891060 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:19.891621 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:19.891624 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:19.891694 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:19.891743 (+ 49us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 19:10:19.906445 (+ 14702us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40165/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 19:10:19.906647 (+ 202us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.9.155.126:44923: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40165/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":89,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.9.155.126:44923: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40165/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260620 19:10:19.908469 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 14959
2026-06-20T19:10:19Z chronyd exiting
I20260620 19:10:19.945269 9837 test_util.cc:175] -----------------------------------------------
I20260620 19:10:19.945406 9837 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982595231230-9837-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (338 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260620 19:10:19.946455 9837 mini_oidc.cc:152] Starting JWKS server
I20260620 19:10:19.947280 9837 webserver.cc:460] Webserver started at https://127.0.0.1:33101/ using document root <none> and password file <none>
I20260620 19:10:19.947374 9837 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 19:10:19.947599 9837 webserver.cc:460] Webserver started at http://127.0.0.1:34061/ using document root <none> and password file <none>
2026-06-20T19:10:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:10:19Z Disabled control of system clock
I20260620 19:10:19.956550 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982595231230-9837-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982595231230-9837-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982595231230-9837-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:40663
--webserver_interface=127.9.155.126
--webserver_port=0
--builtin_ntp_servers=127.9.155.84:42045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.155.126:40663
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33101/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982595231230-9837-0/testchainca.cert with env {}
W20260620 19:10:20.075729 15062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:20.076030 15062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:20.076071 15062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:20.080276 15062 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:10:20.080365 15062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:20.080380 15062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:20.080397 15062 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:10:20.080411 15062 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 19:10:20.080425 15062 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 19:10:20.080438 15062 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33101/jwks.json
I20260620 19:10:20.086282 15062 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:42045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982595231230-9837-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982595231230-9837-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.155.126:40663
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:40663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33101/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.155.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982595231230-9837-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15062
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982595231230-9837-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:10:20.087393 15062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:20.088413 15062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:10:20.096839 15070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:20.096839 15067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:20.096952 15062 server_base.cc:1034] running on GCE node
W20260620 19:10:20.096899 15068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:20.097764 15062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:20.098953 15062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:20.100126 15062 hybrid_clock.cc:648] HybridClock initialized: now 1781982620100083 us; error 59 us; skew 500 ppm
I20260620 19:10:20.102103 15062 webserver.cc:460] Webserver started at http://127.9.155.126:36785/ using document root <none> and password file <none>
I20260620 19:10:20.102651 15062 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:20.102710 15062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:20.102903 15062 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:10:20.105098 15062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982595231230-9837-0/minicluster-data/master-0/data/instance:
uuid: "724ee6dca4474870b1e0d4f73ee5b634"
format_stamp: "Formatted at 2026-06-20 19:10:20 on dist-test-slave-n3fx"
server_key: "22021dc4fbede5739f46225fd6cfde7d"
server_key_iv: "6fb1c7349f97cc4cbadc41abbbe52e50"
server_key_version: "encryptionkey@0"
I20260620 19:10:20.105896 15062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982595231230-9837-0/minicluster-data/master-0/wal/instance:
uuid: "724ee6dca4474870b1e0d4f73ee5b634"
format_stamp: "Formatted at 2026-06-20 19:10:20 on dist-test-slave-n3fx"
server_key: "22021dc4fbede5739f46225fd6cfde7d"
server_key_iv: "6fb1c7349f97cc4cbadc41abbbe52e50"
server_key_version: "encryptionkey@0"
I20260620 19:10:20.110075 15062 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 19:10:20.112784 15076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:20.114116 15062 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:10:20.114223 15062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982595231230-9837-0/minicluster-data/master-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982595231230-9837-0/minicluster-data/master-0/wal
uuid: "724ee6dca4474870b1e0d4f73ee5b634"
format_stamp: "Formatted at 2026-06-20 19:10:20 on dist-test-slave-n3fx"
server_key: "22021dc4fbede5739f46225fd6cfde7d"
server_key_iv: "6fb1c7349f97cc4cbadc41abbbe52e50"
server_key_version: "encryptionkey@0"
I20260620 19:10:20.114378 15062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982595231230-9837-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982595231230-9837-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982595231230-9837-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:20.124118 15062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:20.124879 15062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:20.125048 15062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:20.131467 15062 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.126:40663
I20260620 19:10:20.131481 15128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.126:40663 every 8 connection(s)
I20260620 19:10:20.132426 15062 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
I20260620 19:10:20.132660 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 15062
I20260620 19:10:20.132751 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982595231230-9837-0/minicluster-data/master-0/wal/instance
I20260620 19:10:20.133069 9837 external_mini_cluster.cc:1428] Setting key 082837eed1c7cf59b56c0875fce5f457
I20260620 19:10:20.136050 15129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:20.142211 15129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 724ee6dca4474870b1e0d4f73ee5b634: Bootstrap starting.
I20260620 19:10:20.142360 15132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:20.134516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44100 (local address 127.9.155.126:40663)
0620 19:10:20.135310 (+ 794us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:20.135330 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:20.135369 (+ 39us) server_negotiation.cc:407] Connection header received
0620 19:10:20.136004 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:20.136020 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:20.136471 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:20.136750 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:20.137089 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:20.138034 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:20.139227 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:20.139723 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:20.139839 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:20.140705 (+ 866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:20.140710 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:20.140722 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:10:20.141103 (+ 381us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:20.141153 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:10:20.141172 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:20.141242 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:20.141399 (+ 157us) server_negotiation.cc:299] Negotiation successful
0620 19:10:20.141660 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":142,"threads_started":1}
I20260620 19:10:20.144903 15129 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 724ee6dca4474870b1e0d4f73ee5b634: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:20.145357 9837 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:10:20.145701 15129 log.cc:824] T 00000000000000000000000000000000 P 724ee6dca4474870b1e0d4f73ee5b634: Log is configured to *not* fsync() on all Append() calls
I20260620 19:10:20.147565 15129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 724ee6dca4474870b1e0d4f73ee5b634: No bootstrap required, opened a new log
I20260620 19:10:20.151046 15129 raft_consensus.cc:348] T 00000000000000000000000000000000 P 724ee6dca4474870b1e0d4f73ee5b634 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "724ee6dca4474870b1e0d4f73ee5b634" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 40663 } }
I20260620 19:10:20.151291 15129 raft_consensus.cc:374] T 00000000000000000000000000000000 P 724ee6dca4474870b1e0d4f73ee5b634 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:20.151350 15129 raft_consensus.cc:729] T 00000000000000000000000000000000 P 724ee6dca4474870b1e0d4f73ee5b634 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 724ee6dca4474870b1e0d4f73ee5b634, State: Initialized, Role: FOLLOWER
I20260620 19:10:20.151777 15129 consensus_queue.cc:260] T 00000000000000000000000000000000 P 724ee6dca4474870b1e0d4f73ee5b634 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "724ee6dca4474870b1e0d4f73ee5b634" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 40663 } }
I20260620 19:10:20.151904 15129 raft_consensus.cc:388] T 00000000000000000000000000000000 P 724ee6dca4474870b1e0d4f73ee5b634 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:10:20.151968 15129 raft_consensus.cc:482] T 00000000000000000000000000000000 P 724ee6dca4474870b1e0d4f73ee5b634 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:10:20.152063 15129 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 724ee6dca4474870b1e0d4f73ee5b634 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:20.152988 15129 raft_consensus.cc:504] T 00000000000000000000000000000000 P 724ee6dca4474870b1e0d4f73ee5b634 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "724ee6dca4474870b1e0d4f73ee5b634" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 40663 } }
I20260620 19:10:20.153337 15129 leader_election.cc:302] T 00000000000000000000000000000000 P 724ee6dca4474870b1e0d4f73ee5b634 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 724ee6dca4474870b1e0d4f73ee5b634; no voters:
I20260620 19:10:20.153669 15129 leader_election.cc:288] T 00000000000000000000000000000000 P 724ee6dca4474870b1e0d4f73ee5b634 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:10:20.153810 15134 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 724ee6dca4474870b1e0d4f73ee5b634 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:10:20.154036 15134 raft_consensus.cc:686] T 00000000000000000000000000000000 P 724ee6dca4474870b1e0d4f73ee5b634 [term 1 LEADER]: Becoming Leader. State: Replica: 724ee6dca4474870b1e0d4f73ee5b634, State: Running, Role: LEADER
I20260620 19:10:20.154338 15134 consensus_queue.cc:237] T 00000000000000000000000000000000 P 724ee6dca4474870b1e0d4f73ee5b634 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "724ee6dca4474870b1e0d4f73ee5b634" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 40663 } }
I20260620 19:10:20.154863 15129 sys_catalog.cc:564] T 00000000000000000000000000000000 P 724ee6dca4474870b1e0d4f73ee5b634 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:10:20.155887 15136 sys_catalog.cc:455] T 00000000000000000000000000000000 P 724ee6dca4474870b1e0d4f73ee5b634 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 724ee6dca4474870b1e0d4f73ee5b634. Latest consensus state: current_term: 1 leader_uuid: "724ee6dca4474870b1e0d4f73ee5b634" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "724ee6dca4474870b1e0d4f73ee5b634" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 40663 } } }
I20260620 19:10:20.156011 15136 sys_catalog.cc:458] T 00000000000000000000000000000000 P 724ee6dca4474870b1e0d4f73ee5b634 [sys.catalog]: This master's current role is: LEADER
I20260620 19:10:20.156317 15135 sys_catalog.cc:455] T 00000000000000000000000000000000 P 724ee6dca4474870b1e0d4f73ee5b634 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "724ee6dca4474870b1e0d4f73ee5b634" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "724ee6dca4474870b1e0d4f73ee5b634" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 40663 } } }
I20260620 19:10:20.156423 15135 sys_catalog.cc:458] T 00000000000000000000000000000000 P 724ee6dca4474870b1e0d4f73ee5b634 [sys.catalog]: This master's current role is: LEADER
I20260620 19:10:20.156543 15143 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:10:20.159711 15143 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:10:20.165558 15143 catalog_manager.cc:1347] Generated new cluster ID: a4c259caae9a430d877cde4533a52a2a
I20260620 19:10:20.165648 15143 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:10:20.176693 15143 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:10:20.178030 15143 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:10:20.192163 15143 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 724ee6dca4474870b1e0d4f73ee5b634: Generated new TSK 0
I20260620 19:10:20.193022 15143 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 19:10:20.221774 15159 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260620 19:10:20.221874 15159 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 19:10:20.222651 15132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:20.205800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44110 (local address 127.9.155.126:40663)
0620 19:10:20.205945 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:20.205949 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:20.206059 (+ 110us) server_negotiation.cc:407] Connection header received
0620 19:10:20.206253 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:20.206256 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:20.206320 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:20.206443 (+ 123us) server_negotiation.cc:226] Negotiated authn=JWT
0620 19:10:20.206715 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:20.207527 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:20.208521 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:20.208863 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:20.208966 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:20.209288 (+ 322us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 19:10:20.222169 (+ 12881us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33101/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 19:10:20.222456 (+ 287us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44110: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33101/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":67}
W20260620 19:10:20.222723 15132 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44110: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33101/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260620 19:10:20.222751 15158 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:10:20.205680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40663 (local address 127.0.0.1:44110)
0620 19:10:20.205971 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:20.205984 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:20.206104 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:20.206486 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:20.206488 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:20.206497 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0620 19:10:20.206604 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:20.206609 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:20.207673 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:20.207676 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:20.208416 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:20.208426 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:20.209001 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:20.209005 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:20.209078 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:20.209104 (+ 26us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 19:10:20.222409 (+ 13305us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33101/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 19:10:20.222590 (+ 181us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.9.155.126:40663: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33101/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":107,"threads_started":1}
I20260620 19:10:20.223650 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 15062
2026-06-20T19:10:20Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (315 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260620 19:10:20.262452 9837 mini_oidc.cc:152] Starting JWKS server
I20260620 19:10:20.263351 9837 webserver.cc:460] Webserver started at https://127.0.0.1:45645/ using document root <none> and password file <none>
I20260620 19:10:20.263468 9837 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 19:10:20.263860 9837 webserver.cc:460] Webserver started at http://127.0.0.1:36811/ using document root <none> and password file <none>
2026-06-20T19:10:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:10:20Z Disabled control of system clock
I20260620 19:10:20.272943 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781982595231230-9837-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781982595231230-9837-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781982595231230-9837-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:39105
--webserver_interface=127.9.155.126
--webserver_port=0
--builtin_ntp_servers=127.9.155.84:39715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.155.126:39105
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45645/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 19:10:20.386519 15165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:20.386807 15165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:20.386848 15165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:20.391531 15165 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:10:20.391605 15165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:20.391619 15165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:20.391636 15165 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:10:20.391649 15165 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 19:10:20.391669 15165 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 19:10:20.391702 15165 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45645/jwks.json
I20260620 19:10:20.396854 15165 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:39715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781982595231230-9837-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781982595231230-9837-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.155.126:39105
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:39105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45645/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.155.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15165
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781982595231230-9837-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:10:20.398005 15165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:20.399410 15165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:10:20.407949 15171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:20.407981 15173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:20.407958 15170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:20.408291 15165 server_base.cc:1034] running on GCE node
I20260620 19:10:20.408865 15165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:20.410012 15165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:20.411195 15165 hybrid_clock.cc:648] HybridClock initialized: now 1781982620411172 us; error 41 us; skew 500 ppm
I20260620 19:10:20.413239 15165 webserver.cc:460] Webserver started at http://127.9.155.126:39459/ using document root <none> and password file <none>
I20260620 19:10:20.413803 15165 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:20.413870 15165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:20.414072 15165 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:10:20.416088 15165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781982595231230-9837-0/minicluster-data/master-0/data/instance:
uuid: "3462787092284c54b4f5e38729850bb6"
format_stamp: "Formatted at 2026-06-20 19:10:20 on dist-test-slave-n3fx"
server_key: "8e91b74dfe445d006669c7d38df99623"
server_key_iv: "02fc607a1eddff88884ac33a2b89a183"
server_key_version: "encryptionkey@0"
I20260620 19:10:20.416668 15165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781982595231230-9837-0/minicluster-data/master-0/wal/instance:
uuid: "3462787092284c54b4f5e38729850bb6"
format_stamp: "Formatted at 2026-06-20 19:10:20 on dist-test-slave-n3fx"
server_key: "8e91b74dfe445d006669c7d38df99623"
server_key_iv: "02fc607a1eddff88884ac33a2b89a183"
server_key_version: "encryptionkey@0"
I20260620 19:10:20.420429 15165 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 19:10:20.423076 15179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:20.424496 15165 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:10:20.424633 15165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781982595231230-9837-0/minicluster-data/master-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781982595231230-9837-0/minicluster-data/master-0/wal
uuid: "3462787092284c54b4f5e38729850bb6"
format_stamp: "Formatted at 2026-06-20 19:10:20 on dist-test-slave-n3fx"
server_key: "8e91b74dfe445d006669c7d38df99623"
server_key_iv: "02fc607a1eddff88884ac33a2b89a183"
server_key_version: "encryptionkey@0"
I20260620 19:10:20.424757 15165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781982595231230-9837-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781982595231230-9837-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781982595231230-9837-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:20.435096 15165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:20.435806 15165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:20.435978 15165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:20.442521 15165 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.126:39105
I20260620 19:10:20.442577 15231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.126:39105 every 8 connection(s)
I20260620 19:10:20.443456 15165 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
I20260620 19:10:20.446710 15232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:20.449100 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 15165
I20260620 19:10:20.449215 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781982595231230-9837-0/minicluster-data/master-0/wal/instance
I20260620 19:10:20.449486 9837 external_mini_cluster.cc:1428] Setting key a4bb9d67d46e772a4c43edf9a7d3bc09
I20260620 19:10:20.453006 15232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3462787092284c54b4f5e38729850bb6: Bootstrap starting.
I20260620 19:10:20.455479 15232 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3462787092284c54b4f5e38729850bb6: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:20.456260 15232 log.cc:824] T 00000000000000000000000000000000 P 3462787092284c54b4f5e38729850bb6: Log is configured to *not* fsync() on all Append() calls
I20260620 19:10:20.458338 15232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3462787092284c54b4f5e38729850bb6: No bootstrap required, opened a new log
I20260620 19:10:20.459053 15235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:20.450932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38040 (local address 127.9.155.126:39105)
0620 19:10:20.451658 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:20.451673 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:20.451711 (+ 38us) server_negotiation.cc:407] Connection header received
0620 19:10:20.452484 (+ 773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:20.452505 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:20.452920 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:20.453249 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:20.453623 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:20.454567 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:20.455699 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:20.456237 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:20.456462 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:20.457243 (+ 781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:20.457248 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:20.457260 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:10:20.457641 (+ 381us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:20.457695 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:10:20.457714 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:20.457788 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:20.457959 (+ 171us) server_negotiation.cc:299] Negotiation successful
0620 19:10:20.458232 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":152,"threads_started":1}
I20260620 19:10:20.462191 9837 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:10:20.462245 15232 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3462787092284c54b4f5e38729850bb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3462787092284c54b4f5e38729850bb6" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 39105 } }
I20260620 19:10:20.462534 15232 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3462787092284c54b4f5e38729850bb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:20.462594 15232 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3462787092284c54b4f5e38729850bb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3462787092284c54b4f5e38729850bb6, State: Initialized, Role: FOLLOWER
I20260620 19:10:20.463056 15232 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3462787092284c54b4f5e38729850bb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3462787092284c54b4f5e38729850bb6" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 39105 } }
I20260620 19:10:20.463213 15232 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3462787092284c54b4f5e38729850bb6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:10:20.463284 15232 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3462787092284c54b4f5e38729850bb6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:10:20.463390 15232 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3462787092284c54b4f5e38729850bb6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:20.464524 15232 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3462787092284c54b4f5e38729850bb6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3462787092284c54b4f5e38729850bb6" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 39105 } }
I20260620 19:10:20.464848 15232 leader_election.cc:302] T 00000000000000000000000000000000 P 3462787092284c54b4f5e38729850bb6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3462787092284c54b4f5e38729850bb6; no voters:
I20260620 19:10:20.465118 15232 leader_election.cc:288] T 00000000000000000000000000000000 P 3462787092284c54b4f5e38729850bb6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:10:20.465255 15237 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3462787092284c54b4f5e38729850bb6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:10:20.465492 15237 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3462787092284c54b4f5e38729850bb6 [term 1 LEADER]: Becoming Leader. State: Replica: 3462787092284c54b4f5e38729850bb6, State: Running, Role: LEADER
I20260620 19:10:20.465837 15237 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3462787092284c54b4f5e38729850bb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3462787092284c54b4f5e38729850bb6" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 39105 } }
I20260620 19:10:20.466295 15232 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3462787092284c54b4f5e38729850bb6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:10:20.467373 15239 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3462787092284c54b4f5e38729850bb6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3462787092284c54b4f5e38729850bb6. Latest consensus state: current_term: 1 leader_uuid: "3462787092284c54b4f5e38729850bb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3462787092284c54b4f5e38729850bb6" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 39105 } } }
I20260620 19:10:20.467499 15239 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3462787092284c54b4f5e38729850bb6 [sys.catalog]: This master's current role is: LEADER
I20260620 19:10:20.467808 15238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3462787092284c54b4f5e38729850bb6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3462787092284c54b4f5e38729850bb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3462787092284c54b4f5e38729850bb6" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 39105 } } }
I20260620 19:10:20.467937 15238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3462787092284c54b4f5e38729850bb6 [sys.catalog]: This master's current role is: LEADER
I20260620 19:10:20.467875 15246 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:10:20.470942 15246 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:10:20.476157 15246 catalog_manager.cc:1347] Generated new cluster ID: 5b7c61d8354845c099de96d0099ef6a3
I20260620 19:10:20.476240 15246 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:10:20.486413 15246 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:10:20.487382 15246 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:10:20.491477 15246 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3462787092284c54b4f5e38729850bb6: Generated new TSK 0
I20260620 19:10:20.492164 15246 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:10:20.508294 15235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:20.503000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38044 (local address 127.9.155.126:39105)
0620 19:10:20.503130 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:20.503134 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:20.503274 (+ 140us) server_negotiation.cc:407] Connection header received
0620 19:10:20.503470 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:20.503473 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:20.503532 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:20.503652 (+ 120us) server_negotiation.cc:226] Negotiated authn=JWT
0620 19:10:20.503848 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:20.504694 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:20.505617 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:20.505970 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:20.506068 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:20.506387 (+ 319us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 19:10:20.507874 (+ 1487us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45645/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 19:10:20.508108 (+ 234us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38044: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45645/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":45}
W20260620 19:10:20.508370 15235 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38044: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45645/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260620 19:10:20.509871 15261 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:10:20.502911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:39105 (local address 127.0.0.1:38044)
0620 19:10:20.503196 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:20.503209 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:20.503307 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:20.503649 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:20.503652 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:20.503658 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0620 19:10:20.503760 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:20.503764 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:20.504861 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:20.504864 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:20.505509 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:20.505517 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:20.506101 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:20.506104 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:20.506176 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:20.506201 (+ 25us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 19:10:20.508049 (+ 1848us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45645/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 19:10:20.509697 (+ 1648us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.9.155.126:39105: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45645/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":1463,"thread_start_us":107,"threads_started":1}
I20260620 19:10:20.510229 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 15165
E20260620 19:10:20.518857 15262 webserver.cc:533] Webserver: sslize failed:
E20260620 19:10:20.518941 15262 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-20T19:10:20Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (285 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-20T19:10:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:10:20Z Disabled control of system clock
I20260620 19:10:20.556421 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982595231230-9837-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982595231230-9837-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982595231230-9837-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:46851
--webserver_interface=127.9.155.126
--webserver_port=0
--builtin_ntp_servers=127.9.155.84:36707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.155.126:46851
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982595231230-9837-0/testchainca.cert with env {}
W20260620 19:10:20.668473 15266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:20.668764 15266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:20.668807 15266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:20.672955 15266 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:10:20.673038 15266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:20.673061 15266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:20.673085 15266 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:10:20.673100 15266 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 19:10:20.673116 15266 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 19:10:20.673127 15266 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260620 19:10:20.678814 15266 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:36707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982595231230-9837-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982595231230-9837-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.155.126:46851
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:46851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.155.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982595231230-9837-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15266
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982595231230-9837-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:10:20.679859 15266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:20.680859 15266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:10:20.689566 15274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:20.689577 15272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:20.689568 15271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:20.689760 15266 server_base.cc:1034] running on GCE node
I20260620 19:10:20.690436 15266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:20.691673 15266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:20.692855 15266 hybrid_clock.cc:648] HybridClock initialized: now 1781982620692819 us; error 54 us; skew 500 ppm
I20260620 19:10:20.694792 15266 webserver.cc:460] Webserver started at http://127.9.155.126:35805/ using document root <none> and password file <none>
I20260620 19:10:20.695369 15266 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:20.695447 15266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:20.695648 15266 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:10:20.697782 15266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982595231230-9837-0/minicluster-data/master-0/data/instance:
uuid: "0b6758f9ae634e33b050e70473935fa0"
format_stamp: "Formatted at 2026-06-20 19:10:20 on dist-test-slave-n3fx"
server_key: "38977aac2b29d71a00aedc3ea5bb39ec"
server_key_iv: "df85e2fd4dc77660ac31e96e2b57ccf6"
server_key_version: "encryptionkey@0"
I20260620 19:10:20.698371 15266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982595231230-9837-0/minicluster-data/master-0/wal/instance:
uuid: "0b6758f9ae634e33b050e70473935fa0"
format_stamp: "Formatted at 2026-06-20 19:10:20 on dist-test-slave-n3fx"
server_key: "38977aac2b29d71a00aedc3ea5bb39ec"
server_key_iv: "df85e2fd4dc77660ac31e96e2b57ccf6"
server_key_version: "encryptionkey@0"
I20260620 19:10:20.702311 15266 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 19:10:20.704933 15280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:20.706316 15266 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:10:20.706456 15266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982595231230-9837-0/minicluster-data/master-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982595231230-9837-0/minicluster-data/master-0/wal
uuid: "0b6758f9ae634e33b050e70473935fa0"
format_stamp: "Formatted at 2026-06-20 19:10:20 on dist-test-slave-n3fx"
server_key: "38977aac2b29d71a00aedc3ea5bb39ec"
server_key_iv: "df85e2fd4dc77660ac31e96e2b57ccf6"
server_key_version: "encryptionkey@0"
I20260620 19:10:20.706589 15266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982595231230-9837-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982595231230-9837-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982595231230-9837-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:20.736160 15266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:20.736909 15266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:20.737088 15266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:20.743904 15266 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.126:46851
I20260620 19:10:20.743928 15332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.126:46851 every 8 connection(s)
I20260620 19:10:20.744867 15266 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
I20260620 19:10:20.748085 15333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:20.752530 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 15266
I20260620 19:10:20.752658 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982595231230-9837-0/minicluster-data/master-0/wal/instance
I20260620 19:10:20.752916 9837 external_mini_cluster.cc:1428] Setting key 12bd50860103fd302a84f6148f9113c6
I20260620 19:10:20.754133 15333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b6758f9ae634e33b050e70473935fa0: Bootstrap starting.
I20260620 19:10:20.756574 15333 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b6758f9ae634e33b050e70473935fa0: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:20.757463 15333 log.cc:824] T 00000000000000000000000000000000 P 0b6758f9ae634e33b050e70473935fa0: Log is configured to *not* fsync() on all Append() calls
I20260620 19:10:20.759451 15333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b6758f9ae634e33b050e70473935fa0: No bootstrap required, opened a new log
I20260620 19:10:20.762485 15336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:20.754427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56554 (local address 127.9.155.126:46851)
0620 19:10:20.755101 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:20.755118 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:20.755164 (+ 46us) server_negotiation.cc:407] Connection header received
0620 19:10:20.755856 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:20.755875 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:20.756356 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:20.756681 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:20.757087 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:20.758108 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:20.759049 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:20.759556 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:20.759712 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:20.760678 (+ 966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:20.760684 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:20.760698 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:10:20.761105 (+ 407us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:20.761164 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:10:20.761184 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:20.761283 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:20.761418 (+ 135us) server_negotiation.cc:299] Negotiation successful
0620 19:10:20.761685 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":125,"threads_started":1}
I20260620 19:10:20.763278 15333 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b6758f9ae634e33b050e70473935fa0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b6758f9ae634e33b050e70473935fa0" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 46851 } }
I20260620 19:10:20.763505 15333 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b6758f9ae634e33b050e70473935fa0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:20.763551 15333 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b6758f9ae634e33b050e70473935fa0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6758f9ae634e33b050e70473935fa0, State: Initialized, Role: FOLLOWER
I20260620 19:10:20.763983 15333 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b6758f9ae634e33b050e70473935fa0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b6758f9ae634e33b050e70473935fa0" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 46851 } }
I20260620 19:10:20.764117 15333 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b6758f9ae634e33b050e70473935fa0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:10:20.764187 15333 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b6758f9ae634e33b050e70473935fa0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:10:20.764315 15333 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b6758f9ae634e33b050e70473935fa0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:20.765349 15333 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b6758f9ae634e33b050e70473935fa0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b6758f9ae634e33b050e70473935fa0" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 46851 } }
I20260620 19:10:20.765384 9837 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:10:20.765669 15333 leader_election.cc:302] T 00000000000000000000000000000000 P 0b6758f9ae634e33b050e70473935fa0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b6758f9ae634e33b050e70473935fa0; no voters:
I20260620 19:10:20.765946 15333 leader_election.cc:288] T 00000000000000000000000000000000 P 0b6758f9ae634e33b050e70473935fa0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:10:20.766057 15338 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b6758f9ae634e33b050e70473935fa0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:10:20.766284 15338 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b6758f9ae634e33b050e70473935fa0 [term 1 LEADER]: Becoming Leader. State: Replica: 0b6758f9ae634e33b050e70473935fa0, State: Running, Role: LEADER
I20260620 19:10:20.766606 15338 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b6758f9ae634e33b050e70473935fa0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b6758f9ae634e33b050e70473935fa0" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 46851 } }
I20260620 19:10:20.767198 15333 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b6758f9ae634e33b050e70473935fa0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:10:20.767771 15338 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b6758f9ae634e33b050e70473935fa0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b6758f9ae634e33b050e70473935fa0. Latest consensus state: current_term: 1 leader_uuid: "0b6758f9ae634e33b050e70473935fa0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b6758f9ae634e33b050e70473935fa0" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 46851 } } }
I20260620 19:10:20.767892 15338 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b6758f9ae634e33b050e70473935fa0 [sys.catalog]: This master's current role is: LEADER
I20260620 19:10:20.768296 15340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b6758f9ae634e33b050e70473935fa0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b6758f9ae634e33b050e70473935fa0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b6758f9ae634e33b050e70473935fa0" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 46851 } } }
I20260620 19:10:20.768464 15340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b6758f9ae634e33b050e70473935fa0 [sys.catalog]: This master's current role is: LEADER
I20260620 19:10:20.768613 15344 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:10:20.771762 15344 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:10:20.777504 15344 catalog_manager.cc:1347] Generated new cluster ID: 10bf2dda453f435db7dc9d00e4ed9522
I20260620 19:10:20.777590 15344 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:10:20.790624 15344 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:10:20.791538 15344 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:10:20.797523 15344 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b6758f9ae634e33b050e70473935fa0: Generated new TSK 0
I20260620 19:10:20.798241 15344 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:10:21.231858 15336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:20.808960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56562 (local address 127.9.155.126:46851)
0620 19:10:20.809104 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:20.809108 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:20.809194 (+ 86us) server_negotiation.cc:407] Connection header received
0620 19:10:20.809461 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:20.809464 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:20.809517 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:20.809622 (+ 105us) server_negotiation.cc:226] Negotiated authn=JWT
0620 19:10:20.809879 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:20.810548 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:20.811836 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:20.812064 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:20.812153 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:20.812527 (+ 374us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 19:10:21.231458 (+418931us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 19:10:21.231727 (+ 269us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56562: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":72}
W20260620 19:10:21.231937 15336 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56562: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260620 19:10:21.232048 15362 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:10:20.808813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:46851 (local address 127.0.0.1:56562)
0620 19:10:20.809108 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:20.809122 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:20.809237 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:20.809635 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:20.809638 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:20.809645 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0620 19:10:20.809757 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:20.809763 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:20.810660 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:20.810664 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:20.811707 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:20.811718 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:20.812175 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:20.812179 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:20.812308 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:20.812338 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 19:10:21.231706 (+419368us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 19:10:21.231894 (+ 188us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.9.155.126:46851: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":114,"threads_started":1}
I20260620 19:10:21.232636 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 15266
2026-06-20T19:10:21Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (702 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (122 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (124 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:10:21 dist-test-slave-n3fx krb5kdc[15374](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:10:21 dist-test-slave-n3fx krb5kdc[15374](info): set up 2 sockets
Jun 20 19:10:21 dist-test-slave-n3fx krb5kdc[15374](info): commencing operation
krb5kdc: starting...
W20260620 19:10:22.440526 9837 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/krb5kdc/test-user.keytab.
Jun 20 19:10:22 dist-test-slave-n3fx krb5kdc[15374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:10:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:10:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.155.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.155.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.155.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab.
Entry for principal kudu/127.9.155.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab.
I20260620 19:10:22.558537 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:42691
--webserver_interface=127.9.155.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab
--principal=kudu/127.9.155.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:46657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.155.126:42691
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:10:22.667944 15387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:22.668254 15387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:22.668331 15387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:22.672335 15387 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:10:22.672406 15387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:22.672431 15387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:22.672451 15387 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:10:22.672472 15387 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:10:22.677939 15387 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:46657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.155.126:42691
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab
--principal=kudu/127.9.155.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:42691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.155.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15387
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:10:22.679005 15387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:22.679986 15387 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:10:22.689122 15392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:22.689122 15393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:22.689122 15395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:22.689468 15387 server_base.cc:1034] running on GCE node
I20260620 19:10:22.690076 15387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:22.691262 15387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:22.692461 15387 hybrid_clock.cc:648] HybridClock initialized: now 1781982622692445 us; error 33 us; skew 500 ppm
Jun 20 19:10:22 dist-test-slave-n3fx krb5kdc[15374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982622, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:22.695433 15387 init.cc:377] Logged in from keytab as kudu/127.9.155.126@KRBTEST.COM (short username kudu)
I20260620 19:10:22.696424 15387 webserver.cc:460] Webserver started at http://127.9.155.126:46465/ using document root <none> and password file <none>
I20260620 19:10:22.696960 15387 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:22.697021 15387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:22.697209 15387 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:10:22.699488 15387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/minicluster-data/master-0/data/instance:
uuid: "8e6cc9fe282c44ad8425d17781d22697"
format_stamp: "Formatted at 2026-06-20 19:10:22 on dist-test-slave-n3fx"
server_key: "c6e943aa0cf96b3991a32d5b80f5cf1d"
server_key_iv: "f589af4b491f845d26785a49a451de5a"
server_key_version: "encryptionkey@0"
I20260620 19:10:22.700042 15387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/minicluster-data/master-0/wal/instance:
uuid: "8e6cc9fe282c44ad8425d17781d22697"
format_stamp: "Formatted at 2026-06-20 19:10:22 on dist-test-slave-n3fx"
server_key: "c6e943aa0cf96b3991a32d5b80f5cf1d"
server_key_iv: "f589af4b491f845d26785a49a451de5a"
server_key_version: "encryptionkey@0"
I20260620 19:10:22.703838 15387 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 19:10:22.706403 15402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:22.707640 15387 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 19:10:22.707769 15387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/minicluster-data/master-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/minicluster-data/master-0/wal
uuid: "8e6cc9fe282c44ad8425d17781d22697"
format_stamp: "Formatted at 2026-06-20 19:10:22 on dist-test-slave-n3fx"
server_key: "c6e943aa0cf96b3991a32d5b80f5cf1d"
server_key_iv: "f589af4b491f845d26785a49a451de5a"
server_key_version: "encryptionkey@0"
I20260620 19:10:22.707875 15387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:22.717496 15387 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:22.720611 15387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:22.720790 15387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:22.727551 15387 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.126:42691
I20260620 19:10:22.727644 15454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.126:42691 every 8 connection(s)
I20260620 19:10:22.728547 15387 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
I20260620 19:10:22.731307 15455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:22.734642 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 15387
I20260620 19:10:22.734755 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/minicluster-data/master-0/wal/instance
I20260620 19:10:22.735006 9837 external_mini_cluster.cc:1428] Setting key ecc3698026d34113bb890771aadfe537
I20260620 19:10:22.737131 15455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e6cc9fe282c44ad8425d17781d22697: Bootstrap starting.
I20260620 19:10:22.739809 15455 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e6cc9fe282c44ad8425d17781d22697: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:22.740700 15455 log.cc:824] T 00000000000000000000000000000000 P 8e6cc9fe282c44ad8425d17781d22697: Log is configured to *not* fsync() on all Append() calls
I20260620 19:10:22.742938 15455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e6cc9fe282c44ad8425d17781d22697: No bootstrap required, opened a new log
Jun 20 19:10:22 dist-test-slave-n3fx krb5kdc[15374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:10:22.747656 15455 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e6cc9fe282c44ad8425d17781d22697 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e6cc9fe282c44ad8425d17781d22697" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 42691 } }
I20260620 19:10:22.748008 15455 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e6cc9fe282c44ad8425d17781d22697 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:22.748085 15455 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e6cc9fe282c44ad8425d17781d22697 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e6cc9fe282c44ad8425d17781d22697, State: Initialized, Role: FOLLOWER
I20260620 19:10:22.748656 15455 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e6cc9fe282c44ad8425d17781d22697 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e6cc9fe282c44ad8425d17781d22697" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 42691 } }
I20260620 19:10:22.748826 15455 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e6cc9fe282c44ad8425d17781d22697 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:10:22.748896 15455 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e6cc9fe282c44ad8425d17781d22697 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:10:22.748998 15455 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e6cc9fe282c44ad8425d17781d22697 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:22.750123 15455 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e6cc9fe282c44ad8425d17781d22697 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e6cc9fe282c44ad8425d17781d22697" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 42691 } }
I20260620 19:10:22.750532 15455 leader_election.cc:302] T 00000000000000000000000000000000 P 8e6cc9fe282c44ad8425d17781d22697 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e6cc9fe282c44ad8425d17781d22697; no voters:
I20260620 19:10:22.750870 15455 leader_election.cc:288] T 00000000000000000000000000000000 P 8e6cc9fe282c44ad8425d17781d22697 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:10:22.750983 15460 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e6cc9fe282c44ad8425d17781d22697 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:10:22.751194 15460 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e6cc9fe282c44ad8425d17781d22697 [term 1 LEADER]: Becoming Leader. State: Replica: 8e6cc9fe282c44ad8425d17781d22697, State: Running, Role: LEADER
I20260620 19:10:22.751422 15458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:22.736452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34782 (local address 127.9.155.126:42691)
0620 19:10:22.736883 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:22.736902 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:22.736943 (+ 41us) server_negotiation.cc:407] Connection header received
0620 19:10:22.737719 (+ 776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:22.737742 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:22.738175 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:22.738546 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:22.739260 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:22.740736 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:22.741804 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:22.742467 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:22.742630 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:22.745366 (+ 2736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:22.745373 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:22.745397 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:22.745432 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:22.747910 (+ 2478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:22.748489 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:22.748496 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:22.748502 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:22.748663 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:22.749033 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:22.749036 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:22.749038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:22.749515 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:22.749683 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:22.750049 (+ 366us) server_negotiation.cc:299] Negotiation successful
0620 19:10:22.750391 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":114,"threads_started":1}
I20260620 19:10:22.751838 15460 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e6cc9fe282c44ad8425d17781d22697 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e6cc9fe282c44ad8425d17781d22697" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 42691 } }
I20260620 19:10:22.752206 15455 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e6cc9fe282c44ad8425d17781d22697 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:10:22.753417 15461 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e6cc9fe282c44ad8425d17781d22697 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e6cc9fe282c44ad8425d17781d22697. Latest consensus state: current_term: 1 leader_uuid: "8e6cc9fe282c44ad8425d17781d22697" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e6cc9fe282c44ad8425d17781d22697" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 42691 } } }
I20260620 19:10:22.753775 15461 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e6cc9fe282c44ad8425d17781d22697 [sys.catalog]: This master's current role is: LEADER
I20260620 19:10:22.753684 15462 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e6cc9fe282c44ad8425d17781d22697 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e6cc9fe282c44ad8425d17781d22697" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e6cc9fe282c44ad8425d17781d22697" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 42691 } } }
I20260620 19:10:22.754374 15462 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e6cc9fe282c44ad8425d17781d22697 [sys.catalog]: This master's current role is: LEADER
W20260620 19:10:22.756788 15475 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8e6cc9fe282c44ad8425d17781d22697: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:10:22.756867 15475 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:10:22.756923 15474 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:10:22.757591 15474 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:10:22.764297 15474 catalog_manager.cc:1347] Generated new cluster ID: d24d3935ffcc4feb889e39f9599ff5ef
I20260620 19:10:22.764392 15474 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:10:22.771378 15474 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:10:22.772369 15474 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:10:22.783294 15474 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8e6cc9fe282c44ad8425d17781d22697: Generated new TSK 0
I20260620 19:10:22.784029 15474 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.155.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.155.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.155.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.65.keytab.
Entry for principal kudu/127.9.155.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.65.keytab.
I20260620 19:10:22.810856 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.65:0
--local_ip_for_outbound_sockets=127.9.155.65
--webserver_interface=127.9.155.65
--webserver_port=0
--tserver_master_addrs=127.9.155.126:42691
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.65.keytab
--principal=kudu/127.9.155.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:46657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:10:22.923424 15481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:22.923687 15481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:22.923730 15481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:22.927847 15481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:22.927969 15481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:22.928069 15481 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.65
I20260620 19:10:22.934059 15481 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:46657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.65.keytab
--principal=kudu/127.9.155.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.155.65
--webserver_port=0
--tserver_master_addrs=127.9.155.126:42691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15481
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.65
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:10:22.935109 15481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:22.936077 15481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:10:22.945118 15489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:22.945188 15487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:22.945222 15486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:22.945353 15481 server_base.cc:1034] running on GCE node
I20260620 19:10:22.945710 15481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:22.946336 15481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:22.947505 15481 hybrid_clock.cc:648] HybridClock initialized: now 1781982622947488 us; error 33 us; skew 500 ppm
Jun 20 19:10:22 dist-test-slave-n3fx krb5kdc[15374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982622, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:22.950222 15481 init.cc:377] Logged in from keytab as kudu/127.9.155.65@KRBTEST.COM (short username kudu)
I20260620 19:10:22.951104 15481 webserver.cc:460] Webserver started at http://127.9.155.65:39721/ using document root <none> and password file <none>
I20260620 19:10:22.951650 15481 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:22.951717 15481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:22.951896 15481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:10:22.954113 15481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/minicluster-data/ts-0/data/instance:
uuid: "d8f5ded2c70a4c0086029c64e6cfca81"
format_stamp: "Formatted at 2026-06-20 19:10:22 on dist-test-slave-n3fx"
server_key: "268fb9ed7ebce799321b2d007f245a17"
server_key_iv: "70069d47586ea7ddad7718dff0f6c8fb"
server_key_version: "encryptionkey@0"
I20260620 19:10:22.954658 15481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/minicluster-data/ts-0/wal/instance:
uuid: "d8f5ded2c70a4c0086029c64e6cfca81"
format_stamp: "Formatted at 2026-06-20 19:10:22 on dist-test-slave-n3fx"
server_key: "268fb9ed7ebce799321b2d007f245a17"
server_key_iv: "70069d47586ea7ddad7718dff0f6c8fb"
server_key_version: "encryptionkey@0"
I20260620 19:10:22.958269 15481 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 19:10:22.960738 15496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:22.961927 15481 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:10:22.962035 15481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/minicluster-data/ts-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/minicluster-data/ts-0/wal
uuid: "d8f5ded2c70a4c0086029c64e6cfca81"
format_stamp: "Formatted at 2026-06-20 19:10:22 on dist-test-slave-n3fx"
server_key: "268fb9ed7ebce799321b2d007f245a17"
server_key_iv: "70069d47586ea7ddad7718dff0f6c8fb"
server_key_version: "encryptionkey@0"
I20260620 19:10:22.962131 15481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:22.978739 15481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:22.981799 15481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:22.982003 15481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:22.982591 15481 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:10:22.983407 15481 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:10:22.983457 15481 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:22.983500 15481 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:10:22.983527 15481 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:22.992307 15481 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.65:34837
I20260620 19:10:22.992357 15609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.65:34837 every 8 connection(s)
I20260620 19:10:22.993255 15481 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
I20260620 19:10:22.996997 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 15481
I20260620 19:10:22.997102 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/minicluster-data/ts-0/wal/instance
I20260620 19:10:22.997356 9837 external_mini_cluster.cc:1428] Setting key 0ca593c75496cdb31831072a550e703d
Jun 20 19:10:22 dist-test-slave-n3fx krb5kdc[15374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982622, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.65@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.155.66@KRBTEST.COM; defaulting to no policy
I20260620 19:10:23.007978 15458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:22.995152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.65:39987 (local address 127.9.155.126:42691)
0620 19:10:22.995302 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:22.995306 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:22.996325 (+ 1019us) server_negotiation.cc:407] Connection header received
0620 19:10:22.998164 (+ 1839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:22.998167 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:22.998222 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:22.998286 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:22.999817 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:23.000390 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:23.001541 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:23.001769 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:23.001857 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:23.004340 (+ 2483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:23.004343 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:23.004346 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:23.004372 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:23.005856 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:23.006372 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:23.006375 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:23.006376 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:23.006465 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:23.006802 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:23.006806 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:23.006808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:23.007021 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:23.007108 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:23.007641 (+ 533us) server_negotiation.cc:299] Negotiation successful
0620 19:10:23.007806 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.155.66@KRBTEST.COM" created.
I20260620 19:10:23.008886 15612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:22.995590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:42691 (local address 127.9.155.65:39987)
0620 19:10:22.996132 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:22.996183 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:22.997842 (+ 1659us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:22.998568 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:22.998578 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:22.999117 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:22.999659 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:22.999675 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:23.000517 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:23.000520 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:23.001412 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:23.001422 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:23.001914 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:23.001917 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:23.002095 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:23.002668 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:23.002685 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:23.004151 (+ 1466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:23.005982 (+ 1831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:23.005988 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:23.006006 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:23.006275 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:23.006547 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:23.006550 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:23.006552 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:23.006686 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:23.007118 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:23.007125 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:23.007384 (+ 259us) client_negotiation.cc:769] Sending connection context
0620 19:10:23.007557 (+ 173us) client_negotiation.cc:240] Negotiation successful
0620 19:10:23.007890 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":134,"threads_started":1}
I20260620 19:10:23.010140 15610 heartbeater.cc:347] Connected to a master server at 127.9.155.126:42691
I20260620 19:10:23.010373 15610 heartbeater.cc:464] Registering TS with master...
I20260620 19:10:23.010859 15610 heartbeater.cc:511] Master 127.9.155.126:42691 requested a full tablet report, sending...
I20260620 19:10:23.012691 15419 ts_manager.cc:194] Registered new tserver with Master: d8f5ded2c70a4c0086029c64e6cfca81 (127.9.155.65:34837)
I20260620 19:10:23.014290 15419 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.155.65@KRBTEST.COM'} at 127.9.155.65:39987
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.155.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.66.keytab.
Entry for principal kudu/127.9.155.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.66.keytab.
I20260620 19:10:23.025349 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.66:0
--local_ip_for_outbound_sockets=127.9.155.66
--webserver_interface=127.9.155.66
--webserver_port=0
--tserver_master_addrs=127.9.155.126:42691
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.66.keytab
--principal=kudu/127.9.155.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:46657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:10:23.142961 15615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:23.143318 15615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:23.143396 15615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:23.149354 15615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:23.149451 15615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:23.149544 15615 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.66
I20260620 19:10:23.157006 15615 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:46657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.66.keytab
--principal=kudu/127.9.155.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.155.66
--webserver_port=0
--tserver_master_addrs=127.9.155.126:42691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.66
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:10:23.158460 15615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:23.159844 15615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:10:23.171582 15623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:23.171597 15620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:23.171582 15621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:23.171972 15615 server_base.cc:1034] running on GCE node
I20260620 19:10:23.172420 15615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:23.173092 15615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:23.174258 15615 hybrid_clock.cc:648] HybridClock initialized: now 1781982623174219 us; error 62 us; skew 500 ppm
Jun 20 19:10:23 dist-test-slave-n3fx krb5kdc[15374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982623, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:23.177151 15615 init.cc:377] Logged in from keytab as kudu/127.9.155.66@KRBTEST.COM (short username kudu)
I20260620 19:10:23.178114 15615 webserver.cc:460] Webserver started at http://127.9.155.66:40227/ using document root <none> and password file <none>
I20260620 19:10:23.178716 15615 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:23.178795 15615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:23.179018 15615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:10:23.181223 15615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/minicluster-data/ts-1/data/instance:
uuid: "a60451718ab94fb19ff131bd55462414"
format_stamp: "Formatted at 2026-06-20 19:10:23 on dist-test-slave-n3fx"
server_key: "5e8ea39b64923513aeacf871632d6177"
server_key_iv: "a6c64ce76ce6e23dee777ffd8f92343e"
server_key_version: "encryptionkey@0"
I20260620 19:10:23.181784 15615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/minicluster-data/ts-1/wal/instance:
uuid: "a60451718ab94fb19ff131bd55462414"
format_stamp: "Formatted at 2026-06-20 19:10:23 on dist-test-slave-n3fx"
server_key: "5e8ea39b64923513aeacf871632d6177"
server_key_iv: "a6c64ce76ce6e23dee777ffd8f92343e"
server_key_version: "encryptionkey@0"
I20260620 19:10:23.185720 15615 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 19:10:23.188354 15630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:23.189659 15615 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 19:10:23.189775 15615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/minicluster-data/ts-1/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/minicluster-data/ts-1/wal
uuid: "a60451718ab94fb19ff131bd55462414"
format_stamp: "Formatted at 2026-06-20 19:10:23 on dist-test-slave-n3fx"
server_key: "5e8ea39b64923513aeacf871632d6177"
server_key_iv: "a6c64ce76ce6e23dee777ffd8f92343e"
server_key_version: "encryptionkey@0"
I20260620 19:10:23.189884 15615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:23.209456 15615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:23.213380 15615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:23.213581 15615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:23.214200 15615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:10:23.215091 15615 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:10:23.215144 15615 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:23.215201 15615 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:10:23.215225 15615 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:23.224567 15615 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.66:44725
I20260620 19:10:23.224586 15743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.66:44725 every 8 connection(s)
I20260620 19:10:23.225507 15615 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
I20260620 19:10:23.231992 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 15615
I20260620 19:10:23.232125 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/minicluster-data/ts-1/wal/instance
I20260620 19:10:23.232447 9837 external_mini_cluster.cc:1428] Setting key 74a489b14eb81f398486d25b49074b5d
Jun 20 19:10:23 dist-test-slave-n3fx krb5kdc[15374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982623, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.66@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:10:23.241150 15458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:23.227467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.66:55937 (local address 127.9.155.126:42691)
0620 19:10:23.227638 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:23.227642 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:23.228581 (+ 939us) server_negotiation.cc:407] Connection header received
0620 19:10:23.229983 (+ 1402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:23.229987 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:23.230045 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:23.230117 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:23.231645 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:23.232438 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:23.233951 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:23.234197 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:23.234294 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:23.237198 (+ 2904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:23.237203 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:23.237205 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:23.237240 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:23.238970 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:23.239534 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:23.239537 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:23.239539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:23.239623 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:23.239975 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:23.239979 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:23.239981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:23.240182 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:23.240304 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:23.240834 (+ 530us) server_negotiation.cc:299] Negotiation successful
0620 19:10:23.240982 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
WARNING: no policy specified for kudu/127.9.155.67@KRBTEST.COM; defaulting to no policy
I20260620 19:10:23.242157 15746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:23.227881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:42691 (local address 127.9.155.66:55937)
0620 19:10:23.228405 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:23.228452 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:23.229761 (+ 1309us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:23.230304 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:23.230312 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:23.230884 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:23.231444 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:23.231461 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:23.232656 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:23.232659 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:23.233820 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:23.233833 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:23.234364 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:23.234368 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:23.234560 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:23.235147 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:23.235164 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:23.236962 (+ 1798us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:23.239104 (+ 2142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:23.239112 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:23.239131 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:23.239423 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:23.239707 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:23.239710 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:23.239712 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:23.239893 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:23.240302 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:23.240309 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:23.240599 (+ 290us) client_negotiation.cc:769] Sending connection context
0620 19:10:23.240781 (+ 182us) client_negotiation.cc:240] Negotiation successful
0620 19:10:23.241141 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":122,"threads_started":1}
I20260620 19:10:23.243363 15744 heartbeater.cc:347] Connected to a master server at 127.9.155.126:42691
I20260620 19:10:23.243654 15744 heartbeater.cc:464] Registering TS with master...
I20260620 19:10:23.244159 15744 heartbeater.cc:511] Master 127.9.155.126:42691 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.155.67@KRBTEST.COM" created.
I20260620 19:10:23.245321 15419 ts_manager.cc:194] Registered new tserver with Master: a60451718ab94fb19ff131bd55462414 (127.9.155.66:44725)
I20260620 19:10:23.245950 15419 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.155.66@KRBTEST.COM'} at 127.9.155.66:55937
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.155.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.67.keytab.
Entry for principal kudu/127.9.155.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.67.keytab.
I20260620 19:10:23.260524 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.67:0
--local_ip_for_outbound_sockets=127.9.155.67
--webserver_interface=127.9.155.67
--webserver_port=0
--tserver_master_addrs=127.9.155.126:42691
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.67.keytab
--principal=kudu/127.9.155.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:46657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:10:23.372620 15749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:23.372903 15749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:23.372947 15749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:23.377120 15749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:23.377195 15749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:23.377255 15749 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.67
I20260620 19:10:23.382702 15749 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:46657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.67.keytab
--principal=kudu/127.9.155.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.155.67
--webserver_port=0
--tserver_master_addrs=127.9.155.126:42691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15749
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.67
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:10:23.383824 15749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:23.384863 15749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:10:23.394762 15755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:23.394747 15757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:23.394768 15754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:23.395151 15749 server_base.cc:1034] running on GCE node
I20260620 19:10:23.395578 15749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:23.396323 15749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:23.397508 15749 hybrid_clock.cc:648] HybridClock initialized: now 1781982623397480 us; error 42 us; skew 500 ppm
Jun 20 19:10:23 dist-test-slave-n3fx krb5kdc[15374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982623, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:23.400548 15749 init.cc:377] Logged in from keytab as kudu/127.9.155.67@KRBTEST.COM (short username kudu)
I20260620 19:10:23.401546 15749 webserver.cc:460] Webserver started at http://127.9.155.67:45407/ using document root <none> and password file <none>
I20260620 19:10:23.402096 15749 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:23.402158 15749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:23.402359 15749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:10:23.404525 15749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/minicluster-data/ts-2/data/instance:
uuid: "b12a4dc0484a4a56bfad7bc749582904"
format_stamp: "Formatted at 2026-06-20 19:10:23 on dist-test-slave-n3fx"
server_key: "24ca97c24aa3b201da3603af35e4e303"
server_key_iv: "37885468e20790fdd3f757b275690921"
server_key_version: "encryptionkey@0"
I20260620 19:10:23.405110 15749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/minicluster-data/ts-2/wal/instance:
uuid: "b12a4dc0484a4a56bfad7bc749582904"
format_stamp: "Formatted at 2026-06-20 19:10:23 on dist-test-slave-n3fx"
server_key: "24ca97c24aa3b201da3603af35e4e303"
server_key_iv: "37885468e20790fdd3f757b275690921"
server_key_version: "encryptionkey@0"
I20260620 19:10:23.409313 15749 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 19:10:23.412034 15764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:23.413472 15749 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:10:23.413601 15749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/minicluster-data/ts-2/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/minicluster-data/ts-2/wal
uuid: "b12a4dc0484a4a56bfad7bc749582904"
format_stamp: "Formatted at 2026-06-20 19:10:23 on dist-test-slave-n3fx"
server_key: "24ca97c24aa3b201da3603af35e4e303"
server_key_iv: "37885468e20790fdd3f757b275690921"
server_key_version: "encryptionkey@0"
I20260620 19:10:23.413725 15749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:23.434327 15749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:23.437470 15749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:23.437650 15749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:23.438278 15749 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:10:23.439132 15749 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:10:23.439180 15749 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:23.439220 15749 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:10:23.439235 15749 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:23.448014 15749 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.67:35779
I20260620 19:10:23.448112 15877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.67:35779 every 8 connection(s)
I20260620 19:10:23.448971 15749 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
I20260620 19:10:23.457206 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 15749
I20260620 19:10:23.457312 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/minicluster-data/ts-2/wal/instance
I20260620 19:10:23.457578 9837 external_mini_cluster.cc:1428] Setting key 0ee0bde86089982bf01c29851fcec929
Jun 20 19:10:23 dist-test-slave-n3fx krb5kdc[15374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982623, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.67@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:10:23.463544 15458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:23.450937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.67:53659 (local address 127.9.155.126:42691)
0620 19:10:23.451094 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:23.451099 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:23.451947 (+ 848us) server_negotiation.cc:407] Connection header received
0620 19:10:23.453696 (+ 1749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:23.453700 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:23.453758 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:23.453843 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:23.455025 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:23.455544 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:23.456753 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:23.456973 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:23.457061 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:23.459901 (+ 2840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:23.459904 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:23.459906 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:23.459932 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:23.461426 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:23.461971 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:23.461974 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:23.461975 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:23.462057 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:23.462426 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:23.462429 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:23.462430 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:23.462629 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:23.462722 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:23.463302 (+ 580us) server_negotiation.cc:299] Negotiation successful
0620 19:10:23.463421 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 19:10:23.464423 15880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:23.451302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:42691 (local address 127.9.155.67:53659)
0620 19:10:23.451782 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:23.451832 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:23.453474 (+ 1642us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:23.454032 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:23.454039 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:23.454406 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:23.454881 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:23.454894 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:23.455659 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:23.455662 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:23.456643 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:23.456653 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:23.457113 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:23.457117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:23.457310 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:23.457963 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:23.457981 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:23.459718 (+ 1737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:23.461545 (+ 1827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:23.461551 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:23.461564 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:23.461879 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:23.462160 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:23.462163 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:23.462165 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:23.462317 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:23.462768 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:23.462773 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:23.463048 (+ 275us) client_negotiation.cc:769] Sending connection context
0620 19:10:23.463224 (+ 176us) client_negotiation.cc:240] Negotiation successful
0620 19:10:23.463506 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":118,"threads_started":1}
I20260620 19:10:23.465667 15878 heartbeater.cc:347] Connected to a master server at 127.9.155.126:42691
I20260620 19:10:23.465884 15878 heartbeater.cc:464] Registering TS with master...
I20260620 19:10:23.466418 15878 heartbeater.cc:511] Master 127.9.155.126:42691 requested a full tablet report, sending...
I20260620 19:10:23.467429 15419 ts_manager.cc:194] Registered new tserver with Master: b12a4dc0484a4a56bfad7bc749582904 (127.9.155.67:35779)
I20260620 19:10:23.468004 15419 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.155.67@KRBTEST.COM'} at 127.9.155.67:53659
I20260620 19:10:23.471939 9837 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/krb5kdc/test-admin.keytab.
Jun 20 19:10:23 dist-test-slave-n3fx krb5kdc[15374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:23.489369 9837 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260620 19:10:23.492527 15885 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:10:23.490935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58046 (local address 127.9.155.65:34837)
0620 19:10:23.491292 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:23.491309 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:23.491333 (+ 24us) server_negotiation.cc:407] Connection header received
0620 19:10:23.491389 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:23.491394 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:23.491607 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:23.491747 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:23.492367 (+ 620us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58046: BlockingRecv error: recv got EOF from 127.0.0.1:58046 (error 108)
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":93,"threads_started":1}
W20260620 19:10:23.493016 9837 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/krb5kdc/krb5cc' of type FILE
Jun 20 19:10:23 dist-test-slave-n3fx krb5kdc[15374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:23.494028 9837 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 19:10:23 dist-test-slave-n3fx krb5kdc[15374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:23.494977 9837 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 19:10:23 dist-test-slave-n3fx krb5kdc[15374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.155.65@KRBTEST.COM
I20260620 19:10:23.505546 15885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:23.496010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58058 (local address 127.9.155.65:34837)
0620 19:10:23.496163 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:23.496167 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:23.496258 (+ 91us) server_negotiation.cc:407] Connection header received
0620 19:10:23.496415 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:23.496419 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:23.496465 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:23.496538 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:23.497107 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:23.497949 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:23.498793 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:23.499231 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:23.499328 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:23.501739 (+ 2411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:23.501749 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:23.501764 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:23.501793 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:23.503623 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:23.504159 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:23.504165 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:23.504166 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:23.504303 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:23.504570 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:23.504572 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:23.504573 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:23.504839 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:23.505024 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:23.505210 (+ 186us) server_negotiation.cc:299] Negotiation successful
0620 19:10:23.505363 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260620 19:10:23.509449 15885 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:10:23.508466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58074 (local address 127.9.155.65:34837)
0620 19:10:23.508592 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:23.508596 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:23.508721 (+ 125us) server_negotiation.cc:407] Connection header received
0620 19:10:23.508831 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:23.508833 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:23.508874 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:23.508936 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:23.509345 (+ 409us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58074: BlockingRecv error: recv got EOF from 127.0.0.1:58074 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260620 19:10:23.509590 9837 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/krb5kdc/krb5cc' of type FILE
Jun 20 19:10:23 dist-test-slave-n3fx krb5kdc[15374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:23.510625 9837 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 19:10:23 dist-test-slave-n3fx krb5kdc[15374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:23.511561 9837 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 19:10:23 dist-test-slave-n3fx krb5kdc[15374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.155.65@KRBTEST.COM
I20260620 19:10:23.520789 15885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:23.512524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58086 (local address 127.9.155.65:34837)
0620 19:10:23.512636 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:23.512639 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:23.512686 (+ 47us) server_negotiation.cc:407] Connection header received
0620 19:10:23.512791 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:23.512794 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:23.512831 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:23.512898 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:23.513464 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:23.513999 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:23.514820 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:23.515058 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:23.515141 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:23.517460 (+ 2319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:23.517464 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:23.517488 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:23.517523 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:23.519024 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:23.519464 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:23.519467 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:23.519468 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:23.519541 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:23.519840 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:23.519844 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:23.519845 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:23.520046 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:23.520168 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:23.520507 (+ 339us) server_negotiation.cc:299] Negotiation successful
0620 19:10:23.520626 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
W20260620 19:10:23.523746 15885 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:10:23.522752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58094 (local address 127.9.155.65:34837)
0620 19:10:23.522896 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:23.522901 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:23.522967 (+ 66us) server_negotiation.cc:407] Connection header received
0620 19:10:23.523097 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:23.523100 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:23.523142 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:23.523201 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:23.523634 (+ 433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58094: BlockingRecv error: recv got EOF from 127.0.0.1:58094 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260620 19:10:23.523846 9837 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982595231230-9837-0/krb5kdc/krb5cc' of type FILE
Jun 20 19:10:23 dist-test-slave-n3fx krb5kdc[15374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:23.524942 9837 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 19:10:23 dist-test-slave-n3fx krb5kdc[15374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:23.525751 9837 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 19:10:23 dist-test-slave-n3fx krb5kdc[15374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.155.65@KRBTEST.COM
I20260620 19:10:23.533916 15885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:23.526635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58098 (local address 127.9.155.65:34837)
0620 19:10:23.526761 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:23.526765 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:23.526836 (+ 71us) server_negotiation.cc:407] Connection header received
0620 19:10:23.526997 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:23.527001 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:23.527043 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:23.527106 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:23.527585 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:23.528078 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:23.528895 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:23.529108 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:23.529196 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:23.531233 (+ 2037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:23.531237 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:23.531239 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:23.531261 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:23.532365 (+ 1104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:23.532810 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:23.532814 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:23.532816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:23.532911 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:23.533148 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:23.533151 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:23.533153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:23.533327 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:23.533452 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:23.533657 (+ 205us) server_negotiation.cc:299] Negotiation successful
0620 19:10:23.533762 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 19:10:23.534910 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 15481
I20260620 19:10:23.539980 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 15615
I20260620 19:10:23.544790 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 15749
I20260620 19:10:23.549537 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 15387
2026-06-20T19:10:23Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2071 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:10:23 dist-test-slave-n3fx krb5kdc[15898](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:10:23 dist-test-slave-n3fx krb5kdc[15898](info): set up 2 sockets
Jun 20 19:10:23 dist-test-slave-n3fx krb5kdc[15898](info): commencing operation
krb5kdc: starting...
W20260620 19:10:24.512316 9837 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/krb5kdc/test-user.keytab.
Jun 20 19:10:24 dist-test-slave-n3fx krb5kdc[15898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:10:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:10:24Z Disabled control of system clock
WARNING: no policy specified for oryx/127.9.155.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.155.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.155.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/krb5kdc/oryx_127.9.155.126.keytab.
Entry for principal oryx/127.9.155.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/krb5kdc/oryx_127.9.155.126.keytab.
I20260620 19:10:24.631160 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:44831
--webserver_interface=127.9.155.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/krb5kdc/oryx_127.9.155.126.keytab
--principal=oryx/127.9.155.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:43463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.155.126:44831
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:10:24.741185 15911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:24.741474 15911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:24.741519 15911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:24.745711 15911 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:10:24.745784 15911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:24.745810 15911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:24.745826 15911 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:10:24.745841 15911 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:10:24.751536 15911 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:43463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.155.126:44831
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/krb5kdc/oryx_127.9.155.126.keytab
--principal=oryx/127.9.155.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:44831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.155.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15911
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:10:24.752720 15911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:24.753736 15911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:10:24.762264 15916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:24.762400 15911 server_base.cc:1034] running on GCE node
W20260620 19:10:24.762197 15919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:24.762235 15917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:24.763164 15911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:24.764349 15911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:24.765537 15911 hybrid_clock.cc:648] HybridClock initialized: now 1781982624765513 us; error 45 us; skew 500 ppm
Jun 20 19:10:24 dist-test-slave-n3fx krb5kdc[15898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982624, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.155.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:24.773295 15911 init.cc:377] Logged in from keytab as oryx/127.9.155.126@KRBTEST.COM (short username oryx)
I20260620 19:10:24.774329 15911 webserver.cc:460] Webserver started at http://127.9.155.126:41911/ using document root <none> and password file <none>
I20260620 19:10:24.774887 15911 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:24.774952 15911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:24.775120 15911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:10:24.777285 15911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/master-0/data/instance:
uuid: "6d022764bb4a442e8466cb260239c7e5"
format_stamp: "Formatted at 2026-06-20 19:10:24 on dist-test-slave-n3fx"
server_key: "91014d81f73663446b07ecc72c27301d"
server_key_iv: "37d54e19013037cef5bae22f8f9e85d8"
server_key_version: "encryptionkey@0"
I20260620 19:10:24.777799 15911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/master-0/wal/instance:
uuid: "6d022764bb4a442e8466cb260239c7e5"
format_stamp: "Formatted at 2026-06-20 19:10:24 on dist-test-slave-n3fx"
server_key: "91014d81f73663446b07ecc72c27301d"
server_key_iv: "37d54e19013037cef5bae22f8f9e85d8"
server_key_version: "encryptionkey@0"
I20260620 19:10:24.781556 15911 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 19:10:24.784047 15926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:24.785391 15911 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:10:24.785557 15911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/master-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/master-0/wal
uuid: "6d022764bb4a442e8466cb260239c7e5"
format_stamp: "Formatted at 2026-06-20 19:10:24 on dist-test-slave-n3fx"
server_key: "91014d81f73663446b07ecc72c27301d"
server_key_iv: "37d54e19013037cef5bae22f8f9e85d8"
server_key_version: "encryptionkey@0"
I20260620 19:10:24.785667 15911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:24.813249 15911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:24.816638 15911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:24.816841 15911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:24.823789 15911 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.126:44831
I20260620 19:10:24.824119 15978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.126:44831 every 8 connection(s)
I20260620 19:10:24.824872 15911 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
I20260620 19:10:24.827550 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 15911
I20260620 19:10:24.827680 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/master-0/wal/instance
I20260620 19:10:24.827803 15979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:24.827955 9837 external_mini_cluster.cc:1428] Setting key bb2b67abdd1c496e412dc6ed060d1a37
I20260620 19:10:24.835263 15979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d022764bb4a442e8466cb260239c7e5: Bootstrap starting.
Jun 20 19:10:24 dist-test-slave-n3fx krb5kdc[15898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM
I20260620 19:10:24.837662 15979 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d022764bb4a442e8466cb260239c7e5: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:24.838521 15979 log.cc:824] T 00000000000000000000000000000000 P 6d022764bb4a442e8466cb260239c7e5: Log is configured to *not* fsync() on all Append() calls
I20260620 19:10:24.841005 15979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d022764bb4a442e8466cb260239c7e5: No bootstrap required, opened a new log
I20260620 19:10:24.843087 15982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:24.829428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51964 (local address 127.9.155.126:44831)
0620 19:10:24.829876 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:24.829896 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:24.829941 (+ 45us) server_negotiation.cc:407] Connection header received
0620 19:10:24.830671 (+ 730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:24.830696 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:24.831160 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:24.831557 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:24.832265 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:24.833719 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:24.834638 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:24.835325 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:24.835482 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:24.838052 (+ 2570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:24.838059 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:24.838077 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:24.838112 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:24.840150 (+ 2038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:24.840671 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:24.840677 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:24.840683 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:24.840823 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:24.841115 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:24.841118 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:24.841120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:24.841541 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:24.841732 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:24.842041 (+ 309us) server_negotiation.cc:299] Negotiation successful
0620 19:10:24.842346 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":130,"threads_started":1}
I20260620 19:10:24.844717 15979 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d022764bb4a442e8466cb260239c7e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d022764bb4a442e8466cb260239c7e5" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 44831 } }
I20260620 19:10:24.844972 15979 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d022764bb4a442e8466cb260239c7e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:24.845031 15979 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d022764bb4a442e8466cb260239c7e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d022764bb4a442e8466cb260239c7e5, State: Initialized, Role: FOLLOWER
I20260620 19:10:24.845520 15979 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d022764bb4a442e8466cb260239c7e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d022764bb4a442e8466cb260239c7e5" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 44831 } }
I20260620 19:10:24.845665 15979 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d022764bb4a442e8466cb260239c7e5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:10:24.845733 15979 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d022764bb4a442e8466cb260239c7e5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:10:24.845831 15979 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d022764bb4a442e8466cb260239c7e5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:24.846818 15979 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d022764bb4a442e8466cb260239c7e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d022764bb4a442e8466cb260239c7e5" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 44831 } }
I20260620 19:10:24.847167 15979 leader_election.cc:302] T 00000000000000000000000000000000 P 6d022764bb4a442e8466cb260239c7e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d022764bb4a442e8466cb260239c7e5; no voters:
I20260620 19:10:24.847474 15979 leader_election.cc:288] T 00000000000000000000000000000000 P 6d022764bb4a442e8466cb260239c7e5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:10:24.847636 15984 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d022764bb4a442e8466cb260239c7e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:10:24.847849 15984 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d022764bb4a442e8466cb260239c7e5 [term 1 LEADER]: Becoming Leader. State: Replica: 6d022764bb4a442e8466cb260239c7e5, State: Running, Role: LEADER
I20260620 19:10:24.848140 15984 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d022764bb4a442e8466cb260239c7e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d022764bb4a442e8466cb260239c7e5" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 44831 } }
I20260620 19:10:24.848711 15979 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d022764bb4a442e8466cb260239c7e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:10:24.849880 15986 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d022764bb4a442e8466cb260239c7e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d022764bb4a442e8466cb260239c7e5. Latest consensus state: current_term: 1 leader_uuid: "6d022764bb4a442e8466cb260239c7e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d022764bb4a442e8466cb260239c7e5" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 44831 } } }
I20260620 19:10:24.849887 15985 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d022764bb4a442e8466cb260239c7e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d022764bb4a442e8466cb260239c7e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d022764bb4a442e8466cb260239c7e5" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 44831 } } }
I20260620 19:10:24.850037 15985 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d022764bb4a442e8466cb260239c7e5 [sys.catalog]: This master's current role is: LEADER
I20260620 19:10:24.850037 15986 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d022764bb4a442e8466cb260239c7e5 [sys.catalog]: This master's current role is: LEADER
I20260620 19:10:24.850457 15993 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:10:24.853683 15993 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:10:24.859781 15993 catalog_manager.cc:1347] Generated new cluster ID: 3e1c5ad4f96242d5b1943107518facd5
I20260620 19:10:24.859870 15993 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:10:24.864673 15993 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:10:24.865584 15993 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:10:24.870553 15993 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d022764bb4a442e8466cb260239c7e5: Generated new TSK 0
I20260620 19:10:24.871240 15993 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.9.155.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.155.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.155.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/krb5kdc/oryx_127.9.155.65.keytab.
Entry for principal oryx/127.9.155.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/krb5kdc/oryx_127.9.155.65.keytab.
I20260620 19:10:24.909471 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.65:0
--local_ip_for_outbound_sockets=127.9.155.65
--webserver_interface=127.9.155.65
--webserver_port=0
--tserver_master_addrs=127.9.155.126:44831
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/krb5kdc/oryx_127.9.155.65.keytab
--principal=oryx/127.9.155.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:43463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:10:25.029278 16005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:25.029584 16005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:25.029646 16005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:25.033797 16005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:25.033872 16005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:25.033958 16005 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.65
I20260620 19:10:25.039453 16005 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:43463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/krb5kdc/oryx_127.9.155.65.keytab
--principal=oryx/127.9.155.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.155.65
--webserver_port=0
--tserver_master_addrs=127.9.155.126:44831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16005
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.65
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:10:25.040603 16005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:25.041664 16005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 19:10:25.051702 16005 server_base.cc:1034] running on GCE node
W20260620 19:10:25.051601 16013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:25.051640 16011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:25.051659 16010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:25.052455 16005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:25.053089 16005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:25.054266 16005 hybrid_clock.cc:648] HybridClock initialized: now 1781982625054246 us; error 44 us; skew 500 ppm
Jun 20 19:10:25 dist-test-slave-n3fx krb5kdc[15898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982625, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.155.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:25.057395 16005 init.cc:377] Logged in from keytab as oryx/127.9.155.65@KRBTEST.COM (short username oryx)
I20260620 19:10:25.058420 16005 webserver.cc:460] Webserver started at http://127.9.155.65:37505/ using document root <none> and password file <none>
I20260620 19:10:25.059020 16005 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:25.059092 16005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:25.059293 16005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:10:25.061554 16005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-0/data/instance:
uuid: "af15d4c4a9304cc8a20af15051256305"
format_stamp: "Formatted at 2026-06-20 19:10:25 on dist-test-slave-n3fx"
server_key: "84ebcd940b8416f96e97ca4c5fe018e0"
server_key_iv: "ba1ba47c27c3b058fdead948bcf2b0fc"
server_key_version: "encryptionkey@0"
I20260620 19:10:25.062161 16005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-0/wal/instance:
uuid: "af15d4c4a9304cc8a20af15051256305"
format_stamp: "Formatted at 2026-06-20 19:10:25 on dist-test-slave-n3fx"
server_key: "84ebcd940b8416f96e97ca4c5fe018e0"
server_key_iv: "ba1ba47c27c3b058fdead948bcf2b0fc"
server_key_version: "encryptionkey@0"
I20260620 19:10:25.066098 16005 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260620 19:10:25.068840 16020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:25.070257 16005 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:10:25.070374 16005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-0/wal
uuid: "af15d4c4a9304cc8a20af15051256305"
format_stamp: "Formatted at 2026-06-20 19:10:25 on dist-test-slave-n3fx"
server_key: "84ebcd940b8416f96e97ca4c5fe018e0"
server_key_iv: "ba1ba47c27c3b058fdead948bcf2b0fc"
server_key_version: "encryptionkey@0"
I20260620 19:10:25.070503 16005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:25.106226 16005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:25.110073 16005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:25.110270 16005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:25.110890 16005 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:10:25.111742 16005 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:10:25.111789 16005 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:25.111843 16005 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:10:25.111871 16005 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:25.121268 16005 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.65:42443
I20260620 19:10:25.121308 16133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.65:42443 every 8 connection(s)
I20260620 19:10:25.122182 16005 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
I20260620 19:10:25.127377 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 16005
I20260620 19:10:25.127486 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-0/wal/instance
I20260620 19:10:25.127753 9837 external_mini_cluster.cc:1428] Setting key aec1e7be21ae3cd344bde06675ca32ca
Jun 20 19:10:25 dist-test-slave-n3fx krb5kdc[15898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982625, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.155.65@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM
I20260620 19:10:25.136273 15982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:25.123945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.65:59195 (local address 127.9.155.126:44831)
0620 19:10:25.124082 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:25.124085 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:25.124968 (+ 883us) server_negotiation.cc:407] Connection header received
0620 19:10:25.126194 (+ 1226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:25.126197 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:25.126245 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:25.126328 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:25.127554 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:25.128121 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:25.129478 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:25.129721 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:25.129808 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:25.132496 (+ 2688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:25.132500 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:25.132503 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:25.132531 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:25.134105 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:25.134672 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:25.134675 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:25.134676 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:25.134757 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:25.135103 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:25.135105 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:25.135106 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:25.135293 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:25.135384 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:25.135947 (+ 563us) server_negotiation.cc:299] Negotiation successful
0620 19:10:25.136078 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
WARNING: no policy specified for oryx/127.9.155.66@KRBTEST.COM; defaulting to no policy
I20260620 19:10:25.137197 16136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:25.124324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:44831 (local address 127.9.155.65:59195)
0620 19:10:25.124810 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:25.124854 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:25.125983 (+ 1129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:25.126487 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:25.126494 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:25.126861 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:25.127361 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:25.127375 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:25.128263 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:25.128266 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:25.129346 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:25.129357 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:25.129872 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:25.129876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:25.130063 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:25.130675 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:25.130692 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:25.132285 (+ 1593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:25.134219 (+ 1934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:25.134224 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:25.134238 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:25.134554 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:25.134838 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:25.134848 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:25.134849 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:25.135005 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:25.135405 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:25.135413 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:25.135662 (+ 249us) client_negotiation.cc:769] Sending connection context
0620 19:10:25.135825 (+ 163us) client_negotiation.cc:240] Negotiation successful
0620 19:10:25.136171 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":102,"threads_started":1}
I20260620 19:10:25.138557 16134 heartbeater.cc:347] Connected to a master server at 127.9.155.126:44831
I20260620 19:10:25.138808 16134 heartbeater.cc:464] Registering TS with master...
I20260620 19:10:25.139369 16134 heartbeater.cc:511] Master 127.9.155.126:44831 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.155.66@KRBTEST.COM" created.
I20260620 19:10:25.141110 15943 ts_manager.cc:194] Registered new tserver with Master: af15d4c4a9304cc8a20af15051256305 (127.9.155.65:42443)
I20260620 19:10:25.142300 15943 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.155.65@KRBTEST.COM'} at 127.9.155.65:59195
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.155.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/krb5kdc/oryx_127.9.155.66.keytab.
Entry for principal oryx/127.9.155.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/krb5kdc/oryx_127.9.155.66.keytab.
I20260620 19:10:25.155553 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.66:0
--local_ip_for_outbound_sockets=127.9.155.66
--webserver_interface=127.9.155.66
--webserver_port=0
--tserver_master_addrs=127.9.155.126:44831
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/krb5kdc/oryx_127.9.155.66.keytab
--principal=oryx/127.9.155.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:43463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:10:25.263867 16139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:25.264132 16139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:25.264175 16139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:25.268484 16139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:25.268555 16139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:25.268616 16139 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.66
I20260620 19:10:25.273950 16139 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:43463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/krb5kdc/oryx_127.9.155.66.keytab
--principal=oryx/127.9.155.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.155.66
--webserver_port=0
--tserver_master_addrs=127.9.155.126:44831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16139
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.66
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:10:25.275115 16139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:25.276105 16139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:10:25.285828 16145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:25.285863 16147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:25.285835 16144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:25.285945 16139 server_base.cc:1034] running on GCE node
I20260620 19:10:25.286526 16139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:25.287228 16139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:25.288390 16139 hybrid_clock.cc:648] HybridClock initialized: now 1781982625288371 us; error 33 us; skew 500 ppm
Jun 20 19:10:25 dist-test-slave-n3fx krb5kdc[15898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982625, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.155.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:25.291128 16139 init.cc:377] Logged in from keytab as oryx/127.9.155.66@KRBTEST.COM (short username oryx)
I20260620 19:10:25.292026 16139 webserver.cc:460] Webserver started at http://127.9.155.66:35877/ using document root <none> and password file <none>
I20260620 19:10:25.292605 16139 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:25.292666 16139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:25.292858 16139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:10:25.294903 16139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-1/data/instance:
uuid: "22fadda898e54ffd9a8eaac7a55fcb91"
format_stamp: "Formatted at 2026-06-20 19:10:25 on dist-test-slave-n3fx"
server_key: "c67010f8c0c07fee9aa50373b76b10e6"
server_key_iv: "8879281e625aaadead4c2cfd17490eff"
server_key_version: "encryptionkey@0"
I20260620 19:10:25.295475 16139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-1/wal/instance:
uuid: "22fadda898e54ffd9a8eaac7a55fcb91"
format_stamp: "Formatted at 2026-06-20 19:10:25 on dist-test-slave-n3fx"
server_key: "c67010f8c0c07fee9aa50373b76b10e6"
server_key_iv: "8879281e625aaadead4c2cfd17490eff"
server_key_version: "encryptionkey@0"
I20260620 19:10:25.299365 16139 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 19:10:25.302021 16154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:25.303371 16139 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 19:10:25.303517 16139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-1/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-1/wal
uuid: "22fadda898e54ffd9a8eaac7a55fcb91"
format_stamp: "Formatted at 2026-06-20 19:10:25 on dist-test-slave-n3fx"
server_key: "c67010f8c0c07fee9aa50373b76b10e6"
server_key_iv: "8879281e625aaadead4c2cfd17490eff"
server_key_version: "encryptionkey@0"
I20260620 19:10:25.303625 16139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:25.316154 16139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:25.319690 16139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:25.319890 16139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:25.320537 16139 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:10:25.321393 16139 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:10:25.321441 16139 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:25.321496 16139 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:10:25.321525 16139 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:25.330581 16139 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.66:44711
I20260620 19:10:25.330600 16267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.66:44711 every 8 connection(s)
I20260620 19:10:25.331460 16139 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
Jun 20 19:10:25 dist-test-slave-n3fx krb5kdc[15898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982625, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.155.66@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM
I20260620 19:10:25.341723 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 16139
I20260620 19:10:25.341871 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-1/wal/instance
I20260620 19:10:25.342128 9837 external_mini_cluster.cc:1428] Setting key ec5a3ad2eaea55c4b08f29599d413acc
I20260620 19:10:25.345402 15982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:25.333230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.66:51243 (local address 127.9.155.126:44831)
0620 19:10:25.333346 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:25.333350 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:25.334209 (+ 859us) server_negotiation.cc:407] Connection header received
0620 19:10:25.335437 (+ 1228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:25.335439 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:25.335487 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:25.335558 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:25.336723 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:25.337264 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:25.338416 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:25.338642 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:25.338724 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:25.341359 (+ 2635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:25.341363 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:25.341365 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:25.341390 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:25.343024 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:25.343567 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:25.343570 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:25.343572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:25.343672 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:25.344060 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:25.344063 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:25.344064 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:25.344299 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:25.344437 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:25.345074 (+ 637us) server_negotiation.cc:299] Negotiation successful
0620 19:10:25.345197 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 19:10:25.346335 16270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:25.333629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:44831 (local address 127.9.155.66:51243)
0620 19:10:25.334053 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:25.334095 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:25.335249 (+ 1154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:25.335717 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:25.335723 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:25.336064 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:25.336595 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:25.336610 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:25.337397 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:25.337401 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:25.338295 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:25.338305 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:25.338763 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:25.338765 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:25.338925 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:25.339564 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:25.339581 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:25.341159 (+ 1578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:25.343142 (+ 1983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:25.343148 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:25.343168 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:25.343451 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:25.343770 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:25.343773 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:25.343775 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:25.343964 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:25.344463 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:25.344471 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:25.344810 (+ 339us) client_negotiation.cc:769] Sending connection context
0620 19:10:25.345019 (+ 209us) client_negotiation.cc:240] Negotiation successful
0620 19:10:25.345393 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":103,"threads_started":1}
I20260620 19:10:25.347559 16268 heartbeater.cc:347] Connected to a master server at 127.9.155.126:44831
I20260620 19:10:25.347798 16268 heartbeater.cc:464] Registering TS with master...
I20260620 19:10:25.348376 16268 heartbeater.cc:511] Master 127.9.155.126:44831 requested a full tablet report, sending...
I20260620 19:10:25.349450 15943 ts_manager.cc:194] Registered new tserver with Master: 22fadda898e54ffd9a8eaac7a55fcb91 (127.9.155.66:44711)
I20260620 19:10:25.350263 15943 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.155.66@KRBTEST.COM'} at 127.9.155.66:51243
WARNING: no policy specified for oryx/127.9.155.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.155.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.155.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/krb5kdc/oryx_127.9.155.67.keytab.
Entry for principal oryx/127.9.155.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/krb5kdc/oryx_127.9.155.67.keytab.
I20260620 19:10:25.371330 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.67:0
--local_ip_for_outbound_sockets=127.9.155.67
--webserver_interface=127.9.155.67
--webserver_port=0
--tserver_master_addrs=127.9.155.126:44831
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/krb5kdc/oryx_127.9.155.67.keytab
--principal=oryx/127.9.155.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:43463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:10:25.483444 16273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:25.483702 16273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:25.483747 16273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:25.488154 16273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:25.488247 16273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:25.488334 16273 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.67
I20260620 19:10:25.493819 16273 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:43463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/krb5kdc/oryx_127.9.155.67.keytab
--principal=oryx/127.9.155.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.155.67
--webserver_port=0
--tserver_master_addrs=127.9.155.126:44831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16273
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.67
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:10:25.494927 16273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:25.495941 16273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:10:25.505681 16279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:25.505681 16281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:25.505708 16278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:25.505720 16273 server_base.cc:1034] running on GCE node
I20260620 19:10:25.506311 16273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:25.507009 16273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:25.508330 16273 hybrid_clock.cc:648] HybridClock initialized: now 1781982625508250 us; error 42 us; skew 500 ppm
Jun 20 19:10:25 dist-test-slave-n3fx krb5kdc[15898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982625, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.155.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:25.511082 16273 init.cc:377] Logged in from keytab as oryx/127.9.155.67@KRBTEST.COM (short username oryx)
I20260620 19:10:25.511963 16273 webserver.cc:460] Webserver started at http://127.9.155.67:39105/ using document root <none> and password file <none>
I20260620 19:10:25.512544 16273 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:25.512605 16273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:25.512778 16273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:10:25.514775 16273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-2/data/instance:
uuid: "a2b48205cf584ab18dd060cf238791ab"
format_stamp: "Formatted at 2026-06-20 19:10:25 on dist-test-slave-n3fx"
server_key: "366067e8aeb3ef19a1645e8e27abaedd"
server_key_iv: "e82af44c38030237cfb9a09315afdee8"
server_key_version: "encryptionkey@0"
I20260620 19:10:25.515334 16273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-2/wal/instance:
uuid: "a2b48205cf584ab18dd060cf238791ab"
format_stamp: "Formatted at 2026-06-20 19:10:25 on dist-test-slave-n3fx"
server_key: "366067e8aeb3ef19a1645e8e27abaedd"
server_key_iv: "e82af44c38030237cfb9a09315afdee8"
server_key_version: "encryptionkey@0"
I20260620 19:10:25.519183 16273 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260620 19:10:25.521806 16288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:25.523216 16273 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 19:10:25.523346 16273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-2/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-2/wal
uuid: "a2b48205cf584ab18dd060cf238791ab"
format_stamp: "Formatted at 2026-06-20 19:10:25 on dist-test-slave-n3fx"
server_key: "366067e8aeb3ef19a1645e8e27abaedd"
server_key_iv: "e82af44c38030237cfb9a09315afdee8"
server_key_version: "encryptionkey@0"
I20260620 19:10:25.523469 16273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:25.541898 16273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:25.544919 16273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:25.545095 16273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:25.545697 16273 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:10:25.546547 16273 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:10:25.546595 16273 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:25.546634 16273 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:10:25.546662 16273 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:25.555123 16273 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.67:34277
I20260620 19:10:25.555148 16401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.67:34277 every 8 connection(s)
I20260620 19:10:25.556007 16273 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
I20260620 19:10:25.557883 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 16273
I20260620 19:10:25.558002 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-2/wal/instance
I20260620 19:10:25.558262 9837 external_mini_cluster.cc:1428] Setting key 1c4a4dc28499c5338b4e74a40d8184f7
Jun 20 19:10:25 dist-test-slave-n3fx krb5kdc[15898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982625, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.155.67@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM
I20260620 19:10:25.569684 15982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:25.557746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.67:45493 (local address 127.9.155.126:44831)
0620 19:10:25.557902 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:25.557907 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:25.558779 (+ 872us) server_negotiation.cc:407] Connection header received
0620 19:10:25.560112 (+ 1333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:25.560115 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:25.560160 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:25.560233 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:25.561390 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:25.561918 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:25.563133 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:25.563350 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:25.563427 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:25.565992 (+ 2565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:25.565995 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:25.565997 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:25.566023 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:25.567333 (+ 1310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:25.567864 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:25.567867 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:25.567868 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:25.567949 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:25.568471 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:25.568475 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:25.568477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:25.568688 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:25.568776 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:25.569397 (+ 621us) server_negotiation.cc:299] Negotiation successful
0620 19:10:25.569523 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 19:10:25.570605 16404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:25.558179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:44831 (local address 127.9.155.67:45493)
0620 19:10:25.558630 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:25.558681 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:25.559924 (+ 1243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:25.560450 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:25.560458 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:25.560799 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:25.561261 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:25.561275 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:25.562133 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:25.562136 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:25.563024 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:25.563034 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:25.563517 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:25.563520 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:25.563673 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:25.564277 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:25.564296 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:25.565799 (+ 1503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:25.567460 (+ 1661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:25.567467 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:25.567483 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:25.567762 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:25.568063 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:25.568066 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:25.568068 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:25.568273 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:25.568811 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:25.568817 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:25.569097 (+ 280us) client_negotiation.cc:769] Sending connection context
0620 19:10:25.569272 (+ 175us) client_negotiation.cc:240] Negotiation successful
0620 19:10:25.569688 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":124,"threads_started":1}
I20260620 19:10:25.571732 16402 heartbeater.cc:347] Connected to a master server at 127.9.155.126:44831
I20260620 19:10:25.571964 16402 heartbeater.cc:464] Registering TS with master...
I20260620 19:10:25.572505 16402 heartbeater.cc:511] Master 127.9.155.126:44831 requested a full tablet report, sending...
I20260620 19:10:25.573565 15942 ts_manager.cc:194] Registered new tserver with Master: a2b48205cf584ab18dd060cf238791ab (127.9.155.67:34277)
I20260620 19:10:25.574146 15942 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.155.67@KRBTEST.COM'} at 127.9.155.67:45493
I20260620 19:10:25.584100 9837 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.9.155.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.155.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.155.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/krb5kdc/oryx_127.9.155.125.keytab.
Entry for principal oryx/127.9.155.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/krb5kdc/oryx_127.9.155.125.keytab.
I20260620 19:10:25.609763 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.125:35509
--webserver_interface=127.9.155.125
--webserver_port=0
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/krb5kdc/oryx_127.9.155.125.keytab
--principal=oryx/127.9.155.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:43463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.155.126:44831,127.9.155.125:35509
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:10:25.719195 16407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:25.719470 16407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:25.719511 16407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:25.723783 16407 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:10:25.723872 16407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:25.723896 16407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:25.723914 16407 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:10:25.723928 16407 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:10:25.729489 16407 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:43463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.9.155.126:44831,127.9.155.125:35509
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/krb5kdc/oryx_127.9.155.125.keytab
--principal=oryx/127.9.155.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.125:35509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.9.155.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16407
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
W20260620 19:10:25.729699 16407 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.9.155.126:44831,127.9.155.125:35509'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260620 19:10:25.730548 16407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:25.731494 16407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:10:25.739904 16415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:25.739928 16412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:25.740024 16407 server_base.cc:1034] running on GCE node
W20260620 19:10:25.740283 16413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:25.740844 16407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:25.742017 16407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:25.743196 16407 hybrid_clock.cc:648] HybridClock initialized: now 1781982625743166 us; error 47 us; skew 500 ppm
Jun 20 19:10:25 dist-test-slave-n3fx krb5kdc[15898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982625, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.155.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:25.746124 16407 init.cc:377] Logged in from keytab as oryx/127.9.155.125@KRBTEST.COM (short username oryx)
I20260620 19:10:25.747080 16407 webserver.cc:460] Webserver started at http://127.9.155.125:42603/ using document root <none> and password file <none>
I20260620 19:10:25.747613 16407 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:25.747680 16407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:25.747849 16407 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:10:25.749992 16407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/master-1/data/instance:
uuid: "c20a27597512484cb7546601dada9ca9"
format_stamp: "Formatted at 2026-06-20 19:10:25 on dist-test-slave-n3fx"
server_key: "6d0dbbe89dd212ea53dbca7c95ead41c"
server_key_iv: "fcea16d885cfd4d37819faf24d05243d"
server_key_version: "encryptionkey@0"
I20260620 19:10:25.750530 16407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/master-1/wal/instance:
uuid: "c20a27597512484cb7546601dada9ca9"
format_stamp: "Formatted at 2026-06-20 19:10:25 on dist-test-slave-n3fx"
server_key: "6d0dbbe89dd212ea53dbca7c95ead41c"
server_key_iv: "fcea16d885cfd4d37819faf24d05243d"
server_key_version: "encryptionkey@0"
I20260620 19:10:25.754246 16407 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260620 19:10:25.756860 16422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:25.758145 16407 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:10:25.758249 16407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/master-1/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/master-1/wal
uuid: "c20a27597512484cb7546601dada9ca9"
format_stamp: "Formatted at 2026-06-20 19:10:25 on dist-test-slave-n3fx"
server_key: "6d0dbbe89dd212ea53dbca7c95ead41c"
server_key_iv: "fcea16d885cfd4d37819faf24d05243d"
server_key_version: "encryptionkey@0"
I20260620 19:10:25.758337 16407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:25.772840 16407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:25.776206 16407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:25.776412 16407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:25.783059 16407 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.125:35509
I20260620 19:10:25.783315 16474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.125:35509 every 8 connection(s)
I20260620 19:10:25.784013 16407 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/master-1/data/info.pb
I20260620 19:10:25.786016 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 16407
I20260620 19:10:25.786967 16475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:25.791147 16475 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.155.126" port: 44831 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 19:10:25 dist-test-slave-n3fx krb5kdc[15898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.155.125@KRBTEST.COM
Jun 20 19:10:25 dist-test-slave-n3fx krb5kdc[15898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982625, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.155.125@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM
I20260620 19:10:25.800843 16478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:25.787529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34244 (local address 127.9.155.125:35509)
0620 19:10:25.787910 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:25.787929 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:25.787965 (+ 36us) server_negotiation.cc:407] Connection header received
0620 19:10:25.788666 (+ 701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:25.788690 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:25.789037 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:25.789331 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:25.790253 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:25.791306 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:25.792432 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:25.792903 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:25.793061 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:25.795741 (+ 2680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:25.795748 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:25.795760 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:25.795791 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:25.797652 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:25.798081 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:25.798088 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:25.798094 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:25.798219 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:25.798573 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:25.798575 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:25.798577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:25.799063 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:25.799287 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:25.799622 (+ 335us) server_negotiation.cc:299] Negotiation successful
0620 19:10:25.799881 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":111,"threads_started":1}
I20260620 19:10:25.802456 15982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:25.793096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51974 (local address 127.9.155.126:44831)
0620 19:10:25.793221 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:25.793225 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:25.793587 (+ 362us) server_negotiation.cc:407] Connection header received
0620 19:10:25.793751 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:25.793755 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:25.793802 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:25.793866 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:25.794525 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:25.795286 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:25.796384 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:25.796899 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:25.796986 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:25.799237 (+ 2251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:25.799240 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:25.799243 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:25.799271 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:25.800700 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:25.801213 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:25.801215 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:25.801217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:25.801290 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:25.801559 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:25.801563 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:25.801564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:25.801780 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:25.801869 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:25.802181 (+ 312us) server_negotiation.cc:299] Negotiation successful
0620 19:10:25.802294 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 19:10:25.802474 16479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:25.793014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:44831 (local address 127.0.0.1:51974)
0620 19:10:25.793447 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:25.793500 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:25.793623 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:25.793924 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:25.793930 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:25.794278 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:25.794400 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:25.794406 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:25.795406 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:25.795410 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:25.796273 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:25.796284 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:25.797036 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:25.797040 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:25.797153 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:25.797566 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:25.797581 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:25.799110 (+ 1529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:25.800811 (+ 1701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:25.800817 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:25.800822 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:25.801126 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:25.801370 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:25.801373 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:25.801374 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:25.801463 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:25.801893 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:25.801900 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:25.801999 (+ 99us) client_negotiation.cc:769] Sending connection context
0620 19:10:25.802114 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 19:10:25.802295 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":142,"threads_started":1}
I20260620 19:10:25.803638 16475 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.155.125" port: 35509 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 19:10:25 dist-test-slave-n3fx krb5kdc[15898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982625, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.155.125@KRBTEST.COM for oryx/127.9.155.125@KRBTEST.COM
I20260620 19:10:25.811095 16478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:25.804245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34260 (local address 127.9.155.125:35509)
0620 19:10:25.804369 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:25.804372 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:25.804383 (+ 11us) server_negotiation.cc:407] Connection header received
0620 19:10:25.804411 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:25.804412 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:25.804444 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:25.804505 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:25.804849 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:25.805652 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:25.806383 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:25.806626 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:25.806703 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:25.808547 (+ 1844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:25.808549 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:25.808551 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:25.808569 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:25.809734 (+ 1165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:25.810078 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:25.810081 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:25.810083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:25.810144 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:25.810355 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:25.810358 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:25.810360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:25.810543 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:25.810623 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:25.810832 (+ 209us) server_negotiation.cc:299] Negotiation successful
0620 19:10:25.810962 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":68}
I20260620 19:10:25.811095 16479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:25.804133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.125:35509 (local address 127.0.0.1:34260)
0620 19:10:25.804263 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:25.804274 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:25.804334 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:25.804524 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:25.804526 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:25.804647 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:25.804732 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:25.804737 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:25.805748 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:25.805750 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:25.806280 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:25.806287 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:25.806715 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:25.806717 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:25.806795 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:25.807242 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:25.807252 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:25.808433 (+ 1181us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:25.809825 (+ 1392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:25.809828 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:25.809830 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:25.810003 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:25.810215 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:25.810217 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:25.810219 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:25.810286 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:25.810623 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:25.810626 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:25.810702 (+ 76us) client_negotiation.cc:769] Sending connection context
0620 19:10:25.810782 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 19:10:25.810909 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 19:10:25.813781 16475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c20a27597512484cb7546601dada9ca9: Bootstrap starting.
I20260620 19:10:25.815946 16475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c20a27597512484cb7546601dada9ca9: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:25.816663 16475 log.cc:824] T 00000000000000000000000000000000 P c20a27597512484cb7546601dada9ca9: Log is configured to *not* fsync() on all Append() calls
I20260620 19:10:25.818480 16475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c20a27597512484cb7546601dada9ca9: No bootstrap required, opened a new log
I20260620 19:10:25.821487 16475 raft_consensus.cc:348] T 00000000000000000000000000000000 P c20a27597512484cb7546601dada9ca9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d022764bb4a442e8466cb260239c7e5" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 44831 } } peers { permanent_uuid: "c20a27597512484cb7546601dada9ca9" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 35509 } }
I20260620 19:10:25.821696 16475 raft_consensus.cc:374] T 00000000000000000000000000000000 P c20a27597512484cb7546601dada9ca9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:25.821728 16475 raft_consensus.cc:729] T 00000000000000000000000000000000 P c20a27597512484cb7546601dada9ca9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c20a27597512484cb7546601dada9ca9, State: Initialized, Role: FOLLOWER
I20260620 19:10:25.822163 16475 consensus_queue.cc:260] T 00000000000000000000000000000000 P c20a27597512484cb7546601dada9ca9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d022764bb4a442e8466cb260239c7e5" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 44831 } } peers { permanent_uuid: "c20a27597512484cb7546601dada9ca9" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 35509 } }
I20260620 19:10:25.822669 16481 sys_catalog.cc:455] T 00000000000000000000000000000000 P c20a27597512484cb7546601dada9ca9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d022764bb4a442e8466cb260239c7e5" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 44831 } } peers { permanent_uuid: "c20a27597512484cb7546601dada9ca9" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 35509 } } }
I20260620 19:10:25.822794 16481 sys_catalog.cc:458] T 00000000000000000000000000000000 P c20a27597512484cb7546601dada9ca9 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:10:25.823355 16475 sys_catalog.cc:564] T 00000000000000000000000000000000 P c20a27597512484cb7546601dada9ca9 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 19:10:25.828269 16492 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c20a27597512484cb7546601dada9ca9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:10:25.828397 16492 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:10:25.829862 16407 master_runner.cc:428] Detected that this master c20a27597512484cb7546601dada9ca9 is joining an existing cluster
I20260620 19:10:25.829912 16407 master_runner.cc:432] Initiating AddMaster RPC to add 127.9.155.125:35509
I20260620 19:10:25.838573 15982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:25.832365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51978 (local address 127.9.155.126:44831)
0620 19:10:25.832481 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:25.832485 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:25.832499 (+ 14us) server_negotiation.cc:407] Connection header received
0620 19:10:25.832536 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:25.832539 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:25.832582 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:25.832667 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:25.833080 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:25.833600 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:25.834431 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:25.834657 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:25.834742 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:25.835771 (+ 1029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:25.835775 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:25.835777 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:25.835802 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:25.837112 (+ 1310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:25.837495 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:25.837499 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:25.837501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:25.837572 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:25.837815 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:25.837819 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:25.837820 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:25.837990 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:25.838072 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:25.838310 (+ 238us) server_negotiation.cc:299] Negotiation successful
0620 19:10:25.838412 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 19:10:25.838586 16498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:25.831745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:44831 (local address 127.0.0.1:51978)
0620 19:10:25.832029 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:25.832043 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:25.832124 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:25.832691 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:25.832693 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:25.832860 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:25.832971 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:25.832977 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:25.833693 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:25.833696 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:25.834325 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:25.834333 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:25.834760 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:25.834764 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:25.834865 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:25.835312 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:25.835326 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:25.835647 (+ 321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:25.837210 (+ 1563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:25.837213 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:25.837215 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:25.837409 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:25.837649 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:25.837652 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:25.837654 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:25.837740 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:25.838073 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:25.838076 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:25.838166 (+ 90us) client_negotiation.cc:769] Sending connection context
0620 19:10:25.838250 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 19:10:25.838387 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":94,"threads_started":1}
I20260620 19:10:25.845808 16478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:25.832484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34274 (local address 127.9.155.125:35509)
0620 19:10:25.832573 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:25.832577 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:25.838805 (+ 6228us) server_negotiation.cc:407] Connection header received
0620 19:10:25.838949 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:25.838952 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:25.839001 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:25.839086 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:25.839476 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:25.840246 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:25.841384 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:25.841678 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:25.841761 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:25.842683 (+ 922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:25.842686 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:25.842689 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:25.842709 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:25.844143 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:25.844795 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:25.844799 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:25.844800 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:25.844870 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:25.845073 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:25.845077 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:25.845078 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:25.845236 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:25.845314 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:25.845552 (+ 238us) server_negotiation.cc:299] Negotiation successful
0620 19:10:25.845651 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 19:10:25.846511 16498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:25.832168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.125:35509 (local address 127.0.0.1:34274)
0620 19:10:25.838740 (+ 6572us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:25.838756 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:25.838852 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:25.839110 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:25.839113 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:25.839267 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:25.839371 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:25.839378 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:25.840599 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:25.840602 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:25.841277 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:25.841287 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:25.841778 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:25.841781 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:25.841864 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:25.842217 (+ 353us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:25.842229 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:25.842563 (+ 334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:25.844495 (+ 1932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:25.844498 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:25.844499 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:25.844713 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:25.844945 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:25.844946 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:25.844947 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:25.845003 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:25.845314 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:25.845317 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:25.845394 (+ 77us) client_negotiation.cc:769] Sending connection context
0620 19:10:25.846254 (+ 860us) client_negotiation.cc:240] Negotiation successful
0620 19:10:25.846391 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6509,"thread_start_us":122,"threads_started":1}
Jun 20 19:10:25 dist-test-slave-n3fx krb5kdc[15898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982624, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.155.126@KRBTEST.COM for oryx/127.9.155.125@KRBTEST.COM
I20260620 19:10:25.853396 16501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:25.844313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34278 (local address 127.9.155.125:35509)
0620 19:10:25.844553 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:25.844561 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:25.844573 (+ 12us) server_negotiation.cc:407] Connection header received
0620 19:10:25.844629 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:25.844632 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:25.844679 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:25.844769 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:25.845478 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:25.846237 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:25.847428 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:25.847709 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:25.847785 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:25.850277 (+ 2492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:25.850280 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:25.850282 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:25.850302 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:25.851646 (+ 1344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:25.852142 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:25.852144 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:25.852145 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:25.852262 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:25.852514 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:25.852516 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:25.852517 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:25.852701 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:25.852771 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:25.853106 (+ 335us) server_negotiation.cc:299] Negotiation successful
0620 19:10:25.853229 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":96,"threads_started":1}
I20260620 19:10:25.853457 16500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:25.843741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.125:35509 (local address 127.0.0.1:34278)
0620 19:10:25.844101 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:25.844154 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:25.844297 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:25.844806 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:25.844812 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:25.845228 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:25.845352 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:25.845360 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:25.846348 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:25.846352 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:25.847310 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:25.847321 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:25.847805 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:25.847809 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:25.847918 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:25.848439 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:25.848453 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:25.850150 (+ 1697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:25.851746 (+ 1596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:25.851756 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:25.851763 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:25.852058 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:25.852335 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:25.852338 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:25.852340 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:25.852440 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:25.852785 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:25.852791 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:25.852926 (+ 135us) client_negotiation.cc:769] Sending connection context
0620 19:10:25.853073 (+ 147us) client_negotiation.cc:240] Negotiation successful
0620 19:10:25.853269 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":90,"threads_started":1}
I20260620 19:10:25.854997 15942 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.9.155.125:35509: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "c20a27597512484cb7546601dada9ca9"
member_type: NON_VOTER
last_known_addr {
host: "127.9.155.125"
port: 35509
}
attrs {
promote: true
}
}
dest_uuid: "6d022764bb4a442e8466cb260239c7e5"
cas_config_opid_index: -1
I20260620 19:10:25.855611 15942 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d022764bb4a442e8466cb260239c7e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6d022764bb4a442e8466cb260239c7e5" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 44831 } } peers { permanent_uuid: "c20a27597512484cb7546601dada9ca9" member_type: NON_VOTER last_known_addr { host: "127.9.155.125" port: 35509 } attrs { promote: true } }
I20260620 19:10:25.857544 16503 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6d022764bb4a442e8466cb260239c7e5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER c20a27597512484cb7546601dada9ca9 (127.9.155.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6d022764bb4a442e8466cb260239c7e5" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 44831 } } peers { permanent_uuid: "c20a27597512484cb7546601dada9ca9" member_type: NON_VOTER last_known_addr { host: "127.9.155.125" port: 35509 } attrs { promote: true } } }
I20260620 19:10:25.858196 16449 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c20a27597512484cb7546601dada9ca9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:25.859166 16503 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.9.155.125:35509
I20260620 19:10:25.859516 16449 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c20a27597512484cb7546601dada9ca9 [term 1 FOLLOWER]: Refusing update from remote peer 6d022764bb4a442e8466cb260239c7e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 19:10:25.859493 16503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d022764bb4a442e8466cb260239c7e5 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6d022764bb4a442e8466cb260239c7e5" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6d022764bb4a442e8466cb260239c7e5" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 44831 } } peers { permanent_uuid: "c20a27597512484cb7546601dada9ca9" member_type: NON_VOTER last_known_addr { host: "127.9.155.125" port: 35509 } attrs { promote: true } } }
I20260620 19:10:25.859581 16503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d022764bb4a442e8466cb260239c7e5 [sys.catalog]: This master's current role is: LEADER
I20260620 19:10:25.859584 16407 master.cc:537] Master@127.9.155.125:35509 shutting down...
I20260620 19:10:25.860265 16504 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6d022764bb4a442e8466cb260239c7e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c20a27597512484cb7546601dada9ca9" member_type: NON_VOTER last_known_addr { host: "127.9.155.125" port: 35509 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 19:10:25.862514 15927 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260620 19:10:25.863665 16407 raft_consensus.cc:2229] T 00000000000000000000000000000000 P c20a27597512484cb7546601dada9ca9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:10:25.863934 16407 raft_consensus.cc:2258] T 00000000000000000000000000000000 P c20a27597512484cb7546601dada9ca9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:10:25.864043 16407 tablet_replica.cc:330] T 00000000000000000000000000000000 P c20a27597512484cb7546601dada9ca9: stopping tablet replica
WARNING: no policy specified for oryx/127.9.155.124@KRBTEST.COM; defaulting to no policy
W20260620 19:10:25.867614 15927 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6d022764bb4a442e8466cb260239c7e5 -> Peer c20a27597512484cb7546601dada9ca9 (127.9.155.125:35509): Couldn't send request to peer c20a27597512484cb7546601dada9ca9. Status: Network error: recv got EOF from unknown peer (error 108). This is attempt 1: this message will repeat every 5th retry.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.155.124@KRBTEST.COM" created.
I20260620 19:10:25.869550 16407 master.cc:552] Master@127.9.155.125:35509 shutdown complete.
I20260620 19:10:25.869606 16407 master_runner.cc:305] Clearing existing system tablet
I20260620 19:10:25.869835 16407 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:25.869884 16407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:25.871125 16407 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:10:25.871984 16509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:25.872195 16407 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:25.872399 16407 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:10:25.872462 16407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/master-1/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/master-1/wal
uuid: "c20a27597512484cb7546601dada9ca9"
format_stamp: "Formatted at 2026-06-20 19:10:25 on dist-test-slave-n3fx"
server_key: "6d0dbbe89dd212ea53dbca7c95ead41c"
server_key_iv: "fcea16d885cfd4d37819faf24d05243d"
server_key_version: "encryptionkey@0"
I20260620 19:10:25.875013 16407 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P c20a27597512484cb7546601dada9ca9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:10:25.878345 16407 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P c20a27597512484cb7546601dada9ca9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 19:10:25.878456 16407 log.cc:1196] T 00000000000000000000000000000000 P c20a27597512484cb7546601dada9ca9: Deleting WAL directory at /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260620 19:10:25.878731 16407 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P c20a27597512484cb7546601dada9ca9: Deleting consensus metadata
I20260620 19:10:25.878923 16407 master_runner.cc:315] Copying system tablet from 127.9.155.126:44831
I20260620 19:10:25.880285 16407 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P c20a27597512484cb7546601dada9ca9: tablet copy: Beginning tablet copy session from remote peer at address 127.9.155.126:44831
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.155.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/krb5kdc/oryx_127.9.155.124.keytab.
Entry for principal oryx/127.9.155.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/krb5kdc/oryx_127.9.155.124.keytab.
I20260620 19:10:25.885545 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.124:46355
--webserver_interface=127.9.155.124
--webserver_port=0
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/krb5kdc/oryx_127.9.155.124.keytab
--principal=oryx/127.9.155.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:43463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.155.126:44831,127.9.155.125:35509,127.9.155.124:46355
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:10:25.889057 15982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:25.880948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51990 (local address 127.9.155.126:44831)
0620 19:10:25.881116 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:25.881122 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:25.881206 (+ 84us) server_negotiation.cc:407] Connection header received
0620 19:10:25.881358 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:25.881361 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:25.881412 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:25.881511 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:25.882024 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:25.882908 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:25.883814 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:25.884209 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:25.884343 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:25.885665 (+ 1322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:25.885669 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:25.885672 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:25.885701 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:25.887302 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:25.887709 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:25.887712 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:25.887714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:25.887794 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:25.888110 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:25.888114 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:25.888116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:25.888390 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:25.888494 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:25.888771 (+ 277us) server_negotiation.cc:299] Negotiation successful
0620 19:10:25.888925 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 19:10:25.889333 16515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:25.880878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:44831 (local address 127.0.0.1:51990)
0620 19:10:25.881128 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:25.881142 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:25.881264 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:25.881605 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:25.881607 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:25.881782 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:25.881918 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:25.881926 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:25.883031 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:25.883035 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:25.883706 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:25.883716 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:25.884420 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:25.884424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:25.884504 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:25.885059 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:25.885075 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:25.885501 (+ 426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:25.887410 (+ 1909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:25.887413 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:25.887415 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:25.887618 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:25.887883 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:25.887886 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:25.887888 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:25.887996 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:25.888482 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:25.888485 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:25.888588 (+ 103us) client_negotiation.cc:769] Sending connection context
0620 19:10:25.888713 (+ 125us) client_negotiation.cc:240] Negotiation successful
0620 19:10:25.888873 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":91,"threads_started":1}
I20260620 19:10:25.889916 15963 tablet_copy_service.cc:140] P 6d022764bb4a442e8466cb260239c7e5: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer c20a27597512484cb7546601dada9ca9 ({username='oryx', principal='oryx/127.9.155.125@KRBTEST.COM'} at 127.0.0.1:51990)
I20260620 19:10:25.890115 15963 tablet_copy_service.cc:161] P 6d022764bb4a442e8466cb260239c7e5: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer c20a27597512484cb7546601dada9ca9 at {username='oryx', principal='oryx/127.9.155.125@KRBTEST.COM'} at 127.0.0.1:51990: session id = c20a27597512484cb7546601dada9ca9-00000000000000000000000000000000
I20260620 19:10:25.891594 15963 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 6d022764bb4a442e8466cb260239c7e5: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 19:10:25.893086 16407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 19:10:25.895054 16407 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P c20a27597512484cb7546601dada9ca9: tablet copy: Starting download of 0 data blocks...
I20260620 19:10:25.895236 16407 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P c20a27597512484cb7546601dada9ca9: tablet copy: Starting download of 1 WAL segments...
I20260620 19:10:25.896910 16407 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P c20a27597512484cb7546601dada9ca9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 19:10:25.898563 15963 tablet_copy_service.cc:342] P 6d022764bb4a442e8466cb260239c7e5: Request end of tablet copy session c20a27597512484cb7546601dada9ca9-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.9.155.125@KRBTEST.COM'} at 127.0.0.1:51990
I20260620 19:10:25.898681 15963 tablet_copy_service.cc:433] P 6d022764bb4a442e8466cb260239c7e5: ending tablet copy session c20a27597512484cb7546601dada9ca9-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer c20a27597512484cb7546601dada9ca9
I20260620 19:10:25.905588 16407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:10:25.907258 16517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:25.907382 16518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:25.908388 16520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:25.908473 16407 server_base.cc:1034] running on GCE node
I20260620 19:10:25.912532 16407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:25.912871 16407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:25.914017 16407 hybrid_clock.cc:648] HybridClock initialized: now 1781982625914006 us; error 34 us; skew 500 ppm
Jun 20 19:10:25 dist-test-slave-n3fx krb5kdc[15898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982625, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.155.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:25.915428 16407 init.cc:377] Logged in from keytab as oryx/127.9.155.125@KRBTEST.COM (short username oryx)
I20260620 19:10:25.915882 16407 webserver.cc:460] Webserver started at http://127.9.155.125:35911/ using document root <none> and password file <none>
I20260620 19:10:25.916065 16407 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:25.916097 16407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:25.917222 16407 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:10:25.918087 16527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:25.918298 16407 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:10:25.918375 16407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/master-1/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/master-1/wal
uuid: "c20a27597512484cb7546601dada9ca9"
format_stamp: "Formatted at 2026-06-20 19:10:25 on dist-test-slave-n3fx"
server_key: "6d0dbbe89dd212ea53dbca7c95ead41c"
server_key_iv: "fcea16d885cfd4d37819faf24d05243d"
server_key_version: "encryptionkey@0"
I20260620 19:10:25.918459 16407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:25.927474 16407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:25.929325 16407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:25.933960 16407 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.125:35509
I20260620 19:10:25.934511 16407 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/master-1/data/info.pb
I20260620 19:10:25.934890 16579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.125:35509 every 8 connection(s)
I20260620 19:10:25.936127 16580 sys_catalog.cc:263] Verifying existing consensus state
I20260620 19:10:25.936937 16580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c20a27597512484cb7546601dada9ca9: Bootstrap starting.
I20260620 19:10:25.946053 16580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c20a27597512484cb7546601dada9ca9: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:10:25.946326 16580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c20a27597512484cb7546601dada9ca9: Bootstrap complete.
I20260620 19:10:25.946795 16580 raft_consensus.cc:348] T 00000000000000000000000000000000 P c20a27597512484cb7546601dada9ca9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6d022764bb4a442e8466cb260239c7e5" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 44831 } } peers { permanent_uuid: "c20a27597512484cb7546601dada9ca9" member_type: NON_VOTER last_known_addr { host: "127.9.155.125" port: 35509 } attrs { promote: true } }
I20260620 19:10:25.946903 16580 raft_consensus.cc:729] T 00000000000000000000000000000000 P c20a27597512484cb7546601dada9ca9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c20a27597512484cb7546601dada9ca9, State: Initialized, Role: LEARNER
I20260620 19:10:25.946998 16580 consensus_queue.cc:260] T 00000000000000000000000000000000 P c20a27597512484cb7546601dada9ca9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6d022764bb4a442e8466cb260239c7e5" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 44831 } } peers { permanent_uuid: "c20a27597512484cb7546601dada9ca9" member_type: NON_VOTER last_known_addr { host: "127.9.155.125" port: 35509 } attrs { promote: true } }
I20260620 19:10:25.947278 16583 sys_catalog.cc:455] T 00000000000000000000000000000000 P c20a27597512484cb7546601dada9ca9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6d022764bb4a442e8466cb260239c7e5" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 44831 } } peers { permanent_uuid: "c20a27597512484cb7546601dada9ca9" member_type: NON_VOTER last_known_addr { host: "127.9.155.125" port: 35509 } attrs { promote: true } } }
I20260620 19:10:25.947415 16580 sys_catalog.cc:564] T 00000000000000000000000000000000 P c20a27597512484cb7546601dada9ca9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:10:25.947432 16583 sys_catalog.cc:458] T 00000000000000000000000000000000 P c20a27597512484cb7546601dada9ca9 [sys.catalog]: This master's current role is: LEARNER
I20260620 19:10:25.949837 16594 catalog_manager.cc:1259] Loaded cluster ID: 3e1c5ad4f96242d5b1943107518facd5
I20260620 19:10:25.949939 16594 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c20a27597512484cb7546601dada9ca9: loading cluster ID for follower catalog manager: success
I20260620 19:10:25.951726 16594 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c20a27597512484cb7546601dada9ca9: acquiring CA information for follower catalog manager: success
I20260620 19:10:25.952844 16594 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c20a27597512484cb7546601dada9ca9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 19:10:26.037282 16516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:26.037557 16516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:26.037596 16516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:26.041538 16516 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:10:26.041599 16516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:26.041617 16516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:26.041632 16516 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:10:26.041646 16516 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:10:26.046859 16516 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:43463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.9.155.126:44831,127.9.155.125:35509,127.9.155.124:46355
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/krb5kdc/oryx_127.9.155.124.keytab
--principal=oryx/127.9.155.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.124:46355
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.155.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16516
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:10:26.047894 16516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:26.048983 16516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 19:10:26.057669 16516 server_base.cc:1034] running on GCE node
W20260620 19:10:26.057621 16600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:26.057629 16599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:26.057633 16602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:26.058516 16516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:26.059615 16516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:26.060791 16516 hybrid_clock.cc:648] HybridClock initialized: now 1781982626060774 us; error 34 us; skew 500 ppm
Jun 20 19:10:26 dist-test-slave-n3fx krb5kdc[15898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982626, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.155.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:26.063766 16516 init.cc:377] Logged in from keytab as oryx/127.9.155.124@KRBTEST.COM (short username oryx)
I20260620 19:10:26.064682 16516 webserver.cc:460] Webserver started at http://127.9.155.124:46295/ using document root <none> and password file <none>
I20260620 19:10:26.065212 16516 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:26.065272 16516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:26.065510 16516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:10:26.067610 16516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/master-2/data/instance:
uuid: "12c2fb5fc9eb4fe2850ab1444aee8eda"
format_stamp: "Formatted at 2026-06-20 19:10:26 on dist-test-slave-n3fx"
server_key: "821ad6706390d18ebed33b989f78dc2e"
server_key_iv: "ebd6887241e36b40a11f0c016b5a10d3"
server_key_version: "encryptionkey@0"
I20260620 19:10:26.068153 16516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/master-2/wal/instance:
uuid: "12c2fb5fc9eb4fe2850ab1444aee8eda"
format_stamp: "Formatted at 2026-06-20 19:10:26 on dist-test-slave-n3fx"
server_key: "821ad6706390d18ebed33b989f78dc2e"
server_key_iv: "ebd6887241e36b40a11f0c016b5a10d3"
server_key_version: "encryptionkey@0"
I20260620 19:10:26.071972 16516 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 19:10:26.074549 16609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:26.075815 16516 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 19:10:26.075929 16516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/master-2/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/master-2/wal
uuid: "12c2fb5fc9eb4fe2850ab1444aee8eda"
format_stamp: "Formatted at 2026-06-20 19:10:26 on dist-test-slave-n3fx"
server_key: "821ad6706390d18ebed33b989f78dc2e"
server_key_iv: "ebd6887241e36b40a11f0c016b5a10d3"
server_key_version: "encryptionkey@0"
I20260620 19:10:26.076043 16516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:26.108762 16516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:26.112046 16516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:26.112257 16516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:26.119037 16516 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.124:46355
I20260620 19:10:26.119302 16661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.124:46355 every 8 connection(s)
I20260620 19:10:26.120047 16516 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/master-2/data/info.pb
I20260620 19:10:26.122951 16662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:26.123198 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 16516
I20260620 19:10:26.128123 16662 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.155.126" port: 44831 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 19:10:26 dist-test-slave-n3fx krb5kdc[15898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.155.124@KRBTEST.COM
Jun 20 19:10:26 dist-test-slave-n3fx krb5kdc[15898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982626, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.155.124@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM
I20260620 19:10:26.139106 16664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:26.124957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42890 (local address 127.9.155.124:46355)
0620 19:10:26.125401 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:26.125421 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:26.125465 (+ 44us) server_negotiation.cc:407] Connection header received
0620 19:10:26.126323 (+ 858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:26.126349 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:26.126789 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:26.127191 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:26.128044 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.129526 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.130620 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.131295 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.131449 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:26.133838 (+ 2389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:26.133844 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:26.133855 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:26.133907 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:26.135721 (+ 1814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:26.136132 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:26.136136 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:26.136141 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:26.136302 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:26.136644 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:26.136648 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:26.136649 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:26.137210 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:26.137424 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:26.137806 (+ 382us) server_negotiation.cc:299] Negotiation successful
0620 19:10:26.138102 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"spinlock_wait_cycles":2816,"thread_start_us":132,"threads_started":1}
I20260620 19:10:26.139459 15982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:26.130414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51996 (local address 127.9.155.126:44831)
0620 19:10:26.130552 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:26.130557 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:26.130833 (+ 276us) server_negotiation.cc:407] Connection header received
0620 19:10:26.130980 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:26.130982 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:26.131028 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:26.131092 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:26.131941 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.132499 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.133386 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.133597 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.133684 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:26.136057 (+ 2373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:26.136061 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:26.136063 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:26.136093 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:26.137863 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:26.138326 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:26.138329 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:26.138330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:26.138409 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:26.138652 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:26.138653 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:26.138655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:26.138811 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:26.138877 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:26.139197 (+ 320us) server_negotiation.cc:299] Negotiation successful
0620 19:10:26.139317 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 19:10:26.139495 16665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:26.130256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:44831 (local address 127.0.0.1:51996)
0620 19:10:26.130689 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:26.130737 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:26.130862 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:26.131159 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:26.131165 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:26.131685 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:26.131813 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:26.131819 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.132622 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.132625 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:26.133272 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:26.133281 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.133735 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.133739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:26.133838 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:26.134331 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:26.134343 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:26.135918 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:26.137971 (+ 2053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:26.137978 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:26.137983 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:26.138240 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:26.138490 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:26.138493 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:26.138495 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:26.138577 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:26.138892 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:26.138897 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:26.139015 (+ 118us) client_negotiation.cc:769] Sending connection context
0620 19:10:26.139154 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 19:10:26.139325 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"spinlock_wait_cycles":2304,"thread_start_us":155,"threads_started":1}
I20260620 19:10:26.140468 16662 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.155.125" port: 35509 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:10:26.144712 16134 heartbeater.cc:503] Master 127.9.155.126:44831 was elected leader, sending a full tablet report...
Jun 20 19:10:26 dist-test-slave-n3fx krb5kdc[15898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982626, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.155.124@KRBTEST.COM for oryx/127.9.155.125@KRBTEST.COM
I20260620 19:10:26.148303 16665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:26.140925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.125:35509 (local address 127.0.0.1:34288)
0620 19:10:26.141077 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:26.141090 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:26.141150 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:26.141507 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:26.141510 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:26.141635 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:26.141724 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:26.141729 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.142601 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.142603 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:26.143170 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:26.143178 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.143625 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.143629 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:26.143705 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:26.144088 (+ 383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:26.144098 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:26.145395 (+ 1297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:26.146947 (+ 1552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:26.146950 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:26.146953 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:26.147130 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:26.147388 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:26.147391 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:26.147393 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:26.147467 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:26.147810 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:26.147813 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:26.147906 (+ 93us) client_negotiation.cc:769] Sending connection context
0620 19:10:26.147988 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 19:10:26.148117 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89}
I20260620 19:10:26.148425 16666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:26.141035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34288 (local address 127.9.155.125:35509)
0620 19:10:26.141290 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:26.141294 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:26.141305 (+ 11us) server_negotiation.cc:407] Connection header received
0620 19:10:26.141340 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:26.141342 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:26.141379 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:26.141454 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:26.141830 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.142504 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.143272 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.143510 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.143592 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:26.145541 (+ 1949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:26.145545 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:26.145547 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:26.145571 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:26.146838 (+ 1267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:26.147218 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:26.147221 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:26.147223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:26.147300 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:26.147544 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:26.147547 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:26.147549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:26.147724 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:26.147808 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:26.148072 (+ 264us) server_negotiation.cc:299] Negotiation successful
0620 19:10:26.148229 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":96,"threads_started":1}
I20260620 19:10:26.149019 16662 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.155.124" port: 46355 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 19:10:26 dist-test-slave-n3fx krb5kdc[15898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982626, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.155.124@KRBTEST.COM for oryx/127.9.155.124@KRBTEST.COM
I20260620 19:10:26.156278 16665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:26.149484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.124:46355 (local address 127.0.0.1:42904)
0620 19:10:26.149578 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:26.149591 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:26.149658 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:26.149849 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:26.149851 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:26.149964 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:26.150030 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:26.150034 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.151068 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.151071 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:26.151583 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:26.151591 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.151954 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.151957 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:26.152135 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:26.152535 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:26.152546 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:26.153768 (+ 1222us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:26.155111 (+ 1343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:26.155113 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:26.155114 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:26.155284 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:26.155527 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:26.155529 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:26.155530 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:26.155597 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:26.155865 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:26.155866 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:26.155934 (+ 68us) client_negotiation.cc:769] Sending connection context
0620 19:10:26.156004 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 19:10:26.156104 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":92}
I20260620 19:10:26.156316 16664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:26.149535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42904 (local address 127.9.155.124:46355)
0620 19:10:26.149659 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:26.149663 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:26.149675 (+ 12us) server_negotiation.cc:407] Connection header received
0620 19:10:26.149729 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:26.149732 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:26.149767 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:26.149826 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:26.150107 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.150962 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.151667 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.151880 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.151948 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:26.153899 (+ 1951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:26.153902 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:26.153905 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:26.153924 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:26.155025 (+ 1101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:26.155352 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:26.155354 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:26.155355 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:26.155461 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:26.155656 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:26.155658 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:26.155659 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:26.155803 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:26.155871 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:26.156044 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 19:10:26.156159 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":37}
I20260620 19:10:26.159058 16662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12c2fb5fc9eb4fe2850ab1444aee8eda: Bootstrap starting.
I20260620 19:10:26.161415 16662 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12c2fb5fc9eb4fe2850ab1444aee8eda: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:26.162125 16662 log.cc:824] T 00000000000000000000000000000000 P 12c2fb5fc9eb4fe2850ab1444aee8eda: Log is configured to *not* fsync() on all Append() calls
I20260620 19:10:26.163926 16662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12c2fb5fc9eb4fe2850ab1444aee8eda: No bootstrap required, opened a new log
I20260620 19:10:26.166884 16662 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12c2fb5fc9eb4fe2850ab1444aee8eda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d022764bb4a442e8466cb260239c7e5" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 44831 } } peers { permanent_uuid: "c20a27597512484cb7546601dada9ca9" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 35509 } } peers { permanent_uuid: "12c2fb5fc9eb4fe2850ab1444aee8eda" member_type: VOTER last_known_addr { host: "127.9.155.124" port: 46355 } }
I20260620 19:10:26.167095 16662 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12c2fb5fc9eb4fe2850ab1444aee8eda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:26.167150 16662 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12c2fb5fc9eb4fe2850ab1444aee8eda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12c2fb5fc9eb4fe2850ab1444aee8eda, State: Initialized, Role: FOLLOWER
I20260620 19:10:26.167557 16662 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12c2fb5fc9eb4fe2850ab1444aee8eda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d022764bb4a442e8466cb260239c7e5" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 44831 } } peers { permanent_uuid: "c20a27597512484cb7546601dada9ca9" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 35509 } } peers { permanent_uuid: "12c2fb5fc9eb4fe2850ab1444aee8eda" member_type: VOTER last_known_addr { host: "127.9.155.124" port: 46355 } }
I20260620 19:10:26.167991 16668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12c2fb5fc9eb4fe2850ab1444aee8eda [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d022764bb4a442e8466cb260239c7e5" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 44831 } } peers { permanent_uuid: "c20a27597512484cb7546601dada9ca9" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 35509 } } peers { permanent_uuid: "12c2fb5fc9eb4fe2850ab1444aee8eda" member_type: VOTER last_known_addr { host: "127.9.155.124" port: 46355 } } }
I20260620 19:10:26.168103 16668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12c2fb5fc9eb4fe2850ab1444aee8eda [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:10:26.168753 16662 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12c2fb5fc9eb4fe2850ab1444aee8eda [sys.catalog]: configured and running, proceeding with master startup.
W20260620 19:10:26.173341 16679 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 12c2fb5fc9eb4fe2850ab1444aee8eda: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:10:26.173446 16679 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:10:26.176407 16516 master_runner.cc:428] Detected that this master 12c2fb5fc9eb4fe2850ab1444aee8eda is joining an existing cluster
I20260620 19:10:26.176460 16516 master_runner.cc:432] Initiating AddMaster RPC to add 127.9.155.124:46355
I20260620 19:10:26.187572 15982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:26.178419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52008 (local address 127.9.155.126:44831)
0620 19:10:26.178641 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:26.178645 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:26.178739 (+ 94us) server_negotiation.cc:407] Connection header received
0620 19:10:26.178924 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:26.178928 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:26.178992 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:26.179044 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:26.179740 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.180603 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.181903 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.182251 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.182321 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:26.183690 (+ 1369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:26.183694 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:26.183696 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:26.183725 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:26.185598 (+ 1873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:26.186239 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:26.186243 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:26.186245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:26.186341 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:26.186630 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:26.186633 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:26.186635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:26.186863 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:26.186933 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:26.187271 (+ 338us) server_negotiation.cc:299] Negotiation successful
0620 19:10:26.187393 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137}
I20260620 19:10:26.187577 16664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:26.179017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42912 (local address 127.9.155.124:46355)
0620 19:10:26.179110 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:26.179114 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:26.179624 (+ 510us) server_negotiation.cc:407] Connection header received
0620 19:10:26.179761 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:26.179764 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:26.179797 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:26.179898 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:26.181395 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.182169 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.183326 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.183629 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.183708 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:26.184671 (+ 963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:26.184674 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:26.184677 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:26.184698 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:26.186090 (+ 1392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:26.186480 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:26.186483 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:26.186485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:26.186553 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:26.186775 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:26.186779 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:26.186780 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:26.186961 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:26.187045 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:26.187286 (+ 241us) server_negotiation.cc:299] Negotiation successful
0620 19:10:26.187391 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":36}
I20260620 19:10:26.187597 16687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:26.179190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.124:46355 (local address 127.0.0.1:42912)
0620 19:10:26.179552 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:26.179565 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:26.179675 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:26.180879 (+ 1204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:26.180882 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:26.181155 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:26.181281 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:26.181288 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.182280 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.182283 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:26.183210 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:26.183220 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.183705 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.183708 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:26.183818 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:26.184192 (+ 374us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:26.184205 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:26.184558 (+ 353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:26.186176 (+ 1618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:26.186179 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:26.186181 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:26.186406 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:26.186625 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:26.186628 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:26.186630 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:26.186706 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:26.187045 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:26.187048 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:26.187138 (+ 90us) client_negotiation.cc:769] Sending connection context
0620 19:10:26.187334 (+ 196us) client_negotiation.cc:240] Negotiation successful
0620 19:10:26.187438 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"mutex_wait_us":38,"thread_start_us":136,"threads_started":1}
I20260620 19:10:26.187916 16685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:26.178355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:44831 (local address 127.0.0.1:52008)
0620 19:10:26.178665 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:26.178679 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:26.178809 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:26.179181 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:26.179184 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:26.179460 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:26.179607 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:26.179614 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.180737 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.180740 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:26.181765 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:26.181776 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.182480 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.182484 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:26.182622 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:26.183157 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:26.183168 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:26.183552 (+ 384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:26.185716 (+ 2164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:26.185719 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:26.185721 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:26.186118 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:26.186434 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:26.186436 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:26.186438 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:26.186526 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:26.187001 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:26.187004 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:26.187103 (+ 99us) client_negotiation.cc:769] Sending connection context
0620 19:10:26.187200 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 19:10:26.187766 (+ 566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"mutex_wait_us":82,"thread_start_us":116,"threads_started":1}
I20260620 19:10:26.191133 16668 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12c2fb5fc9eb4fe2850ab1444aee8eda [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:10:26.191284 16668 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12c2fb5fc9eb4fe2850ab1444aee8eda [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d022764bb4a442e8466cb260239c7e5" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 44831 } } peers { permanent_uuid: "c20a27597512484cb7546601dada9ca9" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 35509 } } peers { permanent_uuid: "12c2fb5fc9eb4fe2850ab1444aee8eda" member_type: VOTER last_known_addr { host: "127.9.155.124" port: 46355 } }
I20260620 19:10:26.192235 16668 leader_election.cc:288] T 00000000000000000000000000000000 P 12c2fb5fc9eb4fe2850ab1444aee8eda [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d022764bb4a442e8466cb260239c7e5 (127.9.155.126:44831), c20a27597512484cb7546601dada9ca9 (127.9.155.125:35509)
I20260620 19:10:26.192562 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 16005
I20260620 19:10:26.192624 16554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "12c2fb5fc9eb4fe2850ab1444aee8eda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c20a27597512484cb7546601dada9ca9" is_pre_election: true
I20260620 19:10:26.192803 16554 raft_consensus.cc:1799] T 00000000000000000000000000000000 P c20a27597512484cb7546601dada9ca9 [term 1 LEARNER]: Handling vote request from an unknown peer 12c2fb5fc9eb4fe2850ab1444aee8eda
I20260620 19:10:26.192924 16554 raft_consensus.cc:2396] T 00000000000000000000000000000000 P c20a27597512484cb7546601dada9ca9 [term 1 LEARNER]: Leader pre-election vote request: Denying vote to candidate 12c2fb5fc9eb4fe2850ab1444aee8eda for term 1 because replica has last-logged OpId of term: 1 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260620 19:10:26.193140 15953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "12c2fb5fc9eb4fe2850ab1444aee8eda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d022764bb4a442e8466cb260239c7e5" is_pre_election: true
Jun 20 19:10:26 dist-test-slave-n3fx krb5kdc[15898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982624, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.155.126@KRBTEST.COM for oryx/127.9.155.124@KRBTEST.COM
I20260620 19:10:26.199263 16612 leader_election.cc:302] T 00000000000000000000000000000000 P 12c2fb5fc9eb4fe2850ab1444aee8eda [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 12c2fb5fc9eb4fe2850ab1444aee8eda; no voters: 6d022764bb4a442e8466cb260239c7e5, c20a27597512484cb7546601dada9ca9
I20260620 19:10:26.199569 16668 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12c2fb5fc9eb4fe2850ab1444aee8eda [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:26.200201 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.65:42443
--local_ip_for_outbound_sockets=127.9.155.65
--tserver_master_addrs=127.9.155.126:44831
--webserver_port=37505
--webserver_interface=127.9.155.65
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/krb5kdc/oryx_127.9.155.65.keytab
--principal=oryx/127.9.155.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:43463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.9.155.126:44831,127.9.155.125:35509
--tserver_master_addrs=127.9.155.126:44831,127.9.155.125:35509,127.9.155.124:46355 with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:10:26.202549 16668 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 12c2fb5fc9eb4fe2850ab1444aee8eda [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 19:10:26.205855 16500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:26.191438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.124:46355 (local address 127.0.0.1:42918)
0620 19:10:26.191655 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:26.191668 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:26.191736 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:26.193982 (+ 2246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:26.193986 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:26.194184 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:26.194291 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:26.194297 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.195263 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.195266 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:26.196126 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:26.196136 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.196688 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.196692 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:26.196788 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:26.197303 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:26.197317 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:26.198908 (+ 1591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:26.204331 (+ 5423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:26.204334 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:26.204337 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:26.204586 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:26.204857 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:26.204860 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:26.204862 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:26.204945 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:26.205327 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:26.205330 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:26.205423 (+ 93us) client_negotiation.cc:769] Sending connection context
0620 19:10:26.205512 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 19:10:26.205684 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148}
I20260620 19:10:26.206391 16664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:26.191983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42918 (local address 127.9.155.124:46355)
0620 19:10:26.192076 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:26.192080 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:26.192091 (+ 11us) server_negotiation.cc:407] Connection header received
0620 19:10:26.192148 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:26.192151 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:26.192194 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:26.193948 (+ 1754us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:26.194400 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.195158 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.196264 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.196575 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.196677 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:26.199057 (+ 2380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:26.199060 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:26.199064 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:26.199090 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:26.204153 (+ 5063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:26.204692 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:26.204695 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:26.204697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:26.204771 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:26.205033 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:26.205036 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:26.205038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:26.205227 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:26.205302 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:26.206177 (+ 875us) server_negotiation.cc:299] Negotiation successful
0620 19:10:26.206288 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 19:10:26.206499 16666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:26.178510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34304 (local address 127.9.155.125:35509)
0620 19:10:26.178736 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:26.178740 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:26.178967 (+ 227us) server_negotiation.cc:407] Connection header received
0620 19:10:26.179111 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:26.179114 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:26.179159 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:26.179205 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:26.179805 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.180648 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.182320 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.182651 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.182726 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:26.203351 (+ 20625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:26.203355 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:26.203357 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:26.203384 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:26.204720 (+ 1336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:26.205137 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:26.205141 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:26.205143 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:26.205229 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:26.205533 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:26.205536 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:26.205537 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:26.205754 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:26.205848 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:26.206162 (+ 314us) server_negotiation.cc:299] Negotiation successful
0620 19:10:26.206324 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152}
I20260620 19:10:26.207267 16686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:26.178360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.125:35509 (local address 127.0.0.1:34304)
0620 19:10:26.178900 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:26.178912 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:26.179019 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:26.179374 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:26.179377 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:26.179599 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:26.179696 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:26.179703 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.180827 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.180830 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:26.182200 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:26.182209 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.182866 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.182869 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:26.182952 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:26.202792 (+ 19840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:26.202808 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:26.203185 (+ 377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:26.204830 (+ 1645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:26.204833 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:26.204836 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:26.205035 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:26.205330 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:26.205333 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:26.205335 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:26.205433 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:26.205854 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:26.205857 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:26.205961 (+ 104us) client_negotiation.cc:769] Sending connection context
0620 19:10:26.206879 (+ 918us) client_negotiation.cc:240] Negotiation successful
0620 19:10:26.207091 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"mutex_wait_us":311,"thread_start_us":97,"threads_started":1}
I20260620 19:10:26.207576 15942 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.9.155.124:46355: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "12c2fb5fc9eb4fe2850ab1444aee8eda"
member_type: NON_VOTER
last_known_addr {
host: "127.9.155.124"
port: 46355
}
attrs {
promote: true
}
}
dest_uuid: "6d022764bb4a442e8466cb260239c7e5"
cas_config_opid_index: 5
I20260620 19:10:26.208350 15942 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d022764bb4a442e8466cb260239c7e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6d022764bb4a442e8466cb260239c7e5" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 44831 } } peers { permanent_uuid: "c20a27597512484cb7546601dada9ca9" member_type: NON_VOTER last_known_addr { host: "127.9.155.125" port: 35509 } attrs { promote: true } } peers { permanent_uuid: "12c2fb5fc9eb4fe2850ab1444aee8eda" member_type: NON_VOTER last_known_addr { host: "127.9.155.124" port: 46355 } attrs { promote: true } }
I20260620 19:10:26.209924 16504 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6d022764bb4a442e8466cb260239c7e5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 12c2fb5fc9eb4fe2850ab1444aee8eda (127.9.155.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6d022764bb4a442e8466cb260239c7e5" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 44831 } } peers { permanent_uuid: "c20a27597512484cb7546601dada9ca9" member_type: NON_VOTER last_known_addr { host: "127.9.155.125" port: 35509 } attrs { promote: true } } peers { permanent_uuid: "12c2fb5fc9eb4fe2850ab1444aee8eda" member_type: NON_VOTER last_known_addr { host: "127.9.155.124" port: 46355 } attrs { promote: true } } }
I20260620 19:10:26.210572 16636 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 12c2fb5fc9eb4fe2850ab1444aee8eda [term 1 FOLLOWER]: Refusing update from remote peer 6d022764bb4a442e8466cb260239c7e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 19:10:26.211447 16504 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.9.155.124:46355
I20260620 19:10:26.211625 16504 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6d022764bb4a442e8466cb260239c7e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12c2fb5fc9eb4fe2850ab1444aee8eda" member_type: NON_VOTER last_known_addr { host: "127.9.155.124" port: 46355 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:10:26.212316 16503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d022764bb4a442e8466cb260239c7e5 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6d022764bb4a442e8466cb260239c7e5" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6d022764bb4a442e8466cb260239c7e5" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 44831 } } peers { permanent_uuid: "c20a27597512484cb7546601dada9ca9" member_type: NON_VOTER last_known_addr { host: "127.9.155.125" port: 35509 } attrs { promote: true } } peers { permanent_uuid: "12c2fb5fc9eb4fe2850ab1444aee8eda" member_type: NON_VOTER last_known_addr { host: "127.9.155.124" port: 46355 } attrs { promote: true } } }
I20260620 19:10:26.212445 16503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d022764bb4a442e8466cb260239c7e5 [sys.catalog]: This master's current role is: LEADER
I20260620 19:10:26.214308 16516 master.cc:537] Master@127.9.155.124:46355 shutting down...
W20260620 19:10:26.215956 15930 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6d022764bb4a442e8466cb260239c7e5 -> Peer 12c2fb5fc9eb4fe2850ab1444aee8eda (127.9.155.124:46355): Couldn't send request to peer 12c2fb5fc9eb4fe2850ab1444aee8eda. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 19:10:26.220444 16516 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 12c2fb5fc9eb4fe2850ab1444aee8eda [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:10:26.220752 16516 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 12c2fb5fc9eb4fe2850ab1444aee8eda [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:10:26.220839 16516 tablet_replica.cc:330] T 00000000000000000000000000000000 P 12c2fb5fc9eb4fe2850ab1444aee8eda: stopping tablet replica
I20260620 19:10:26.220978 16666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:26.212995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34316 (local address 127.9.155.125:35509)
0620 19:10:26.214341 (+ 1346us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:26.214346 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:26.214359 (+ 13us) server_negotiation.cc:407] Connection header received
0620 19:10:26.214415 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:26.214419 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:26.214476 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:26.214566 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:10:26.216627 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.217454 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.219290 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.220004 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.220083 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:26.220264 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:26.220762 (+ 498us) server_negotiation.cc:299] Negotiation successful
0620 19:10:26.220847 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1259}
I20260620 19:10:26.221374 16500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:26.212741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.125:35509 (local address 127.0.0.1:34316)
0620 19:10:26.212862 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:26.212877 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:26.212969 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:26.216339 (+ 3370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:26.216343 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:26.216366 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:10:26.216487 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:26.216494 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.217696 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.217700 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:26.219155 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:26.219166 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.220497 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.220500 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:26.220604 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:26.220652 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 19:10:26.221159 (+ 507us) client_negotiation.cc:240] Negotiation successful
0620 19:10:26.221251 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 19:10:26.221894 16554 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c20a27597512484cb7546601dada9ca9 [term 1 LEARNER]: Refusing update from remote peer 6d022764bb4a442e8466cb260239c7e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 19:10:26.222357 16689 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6d022764bb4a442e8466cb260239c7e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c20a27597512484cb7546601dada9ca9" member_type: NON_VOTER last_known_addr { host: "127.9.155.125" port: 35509 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 19:10:26.224189 16554 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c20a27597512484cb7546601dada9ca9 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 12c2fb5fc9eb4fe2850ab1444aee8eda (127.9.155.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6d022764bb4a442e8466cb260239c7e5" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 44831 } } peers { permanent_uuid: "c20a27597512484cb7546601dada9ca9" member_type: NON_VOTER last_known_addr { host: "127.9.155.125" port: 35509 } attrs { promote: true } } peers { permanent_uuid: "12c2fb5fc9eb4fe2850ab1444aee8eda" member_type: NON_VOTER last_known_addr { host: "127.9.155.124" port: 46355 } attrs { promote: true } } }
I20260620 19:10:26.227451 16583 sys_catalog.cc:455] T 00000000000000000000000000000000 P c20a27597512484cb7546601dada9ca9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d022764bb4a442e8466cb260239c7e5. Latest consensus state: current_term: 1 leader_uuid: "6d022764bb4a442e8466cb260239c7e5" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6d022764bb4a442e8466cb260239c7e5" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 44831 } } peers { permanent_uuid: "c20a27597512484cb7546601dada9ca9" member_type: NON_VOTER last_known_addr { host: "127.9.155.125" port: 35509 } attrs { promote: true } } peers { permanent_uuid: "12c2fb5fc9eb4fe2850ab1444aee8eda" member_type: NON_VOTER last_known_addr { host: "127.9.155.124" port: 46355 } attrs { promote: true } } }
I20260620 19:10:26.227583 16583 sys_catalog.cc:458] T 00000000000000000000000000000000 P c20a27597512484cb7546601dada9ca9 [sys.catalog]: This master's current role is: LEARNER
I20260620 19:10:26.228168 16691 sys_catalog.cc:455] T 00000000000000000000000000000000 P c20a27597512484cb7546601dada9ca9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6d022764bb4a442e8466cb260239c7e5" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6d022764bb4a442e8466cb260239c7e5" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 44831 } } peers { permanent_uuid: "c20a27597512484cb7546601dada9ca9" member_type: NON_VOTER last_known_addr { host: "127.9.155.125" port: 35509 } attrs { promote: true } } peers { permanent_uuid: "12c2fb5fc9eb4fe2850ab1444aee8eda" member_type: NON_VOTER last_known_addr { host: "127.9.155.124" port: 46355 } attrs { promote: true } } }
I20260620 19:10:26.228380 16691 sys_catalog.cc:458] T 00000000000000000000000000000000 P c20a27597512484cb7546601dada9ca9 [sys.catalog]: This master's current role is: LEARNER
I20260620 19:10:26.228374 16504 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 6d022764bb4a442e8466cb260239c7e5: attempting to promote NON_VOTER c20a27597512484cb7546601dada9ca9 to VOTER
I20260620 19:10:26.228842 16504 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d022764bb4a442e8466cb260239c7e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6d022764bb4a442e8466cb260239c7e5" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 44831 } } peers { permanent_uuid: "c20a27597512484cb7546601dada9ca9" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 35509 } attrs { promote: false } } peers { permanent_uuid: "12c2fb5fc9eb4fe2850ab1444aee8eda" member_type: NON_VOTER last_known_addr { host: "127.9.155.124" port: 46355 } attrs { promote: true } }
I20260620 19:10:26.230594 16554 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c20a27597512484cb7546601dada9ca9 [term 1 LEARNER]: Refusing update from remote peer 6d022764bb4a442e8466cb260239c7e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260620 19:10:26.230621 16500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:26.230217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.124:46355 (local address 127.0.0.1:42934)
0620 19:10:26.230396 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:26.230538 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.124:46355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260620 19:10:26.231138 16692 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6d022764bb4a442e8466cb260239c7e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c20a27597512484cb7546601dada9ca9" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 35509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 19:10:26.231429 16500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:26.231033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.124:46355 (local address 127.0.0.1:42946)
0620 19:10:26.231294 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:26.231339 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.124:46355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":93,"threads_started":1}
W20260620 19:10:26.231508 15930 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6d022764bb4a442e8466cb260239c7e5 -> Peer 12c2fb5fc9eb4fe2850ab1444aee8eda (127.9.155.124:46355): Couldn't send request to peer 12c2fb5fc9eb4fe2850ab1444aee8eda. Status: Network error: Client connection negotiation failed: client connection to 127.9.155.124:46355: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 19:10:26.232991 16689 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6d022764bb4a442e8466cb260239c7e5 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c20a27597512484cb7546601dada9ca9 (127.9.155.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6d022764bb4a442e8466cb260239c7e5" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 44831 } } peers { permanent_uuid: "c20a27597512484cb7546601dada9ca9" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 35509 } attrs { promote: false } } peers { permanent_uuid: "12c2fb5fc9eb4fe2850ab1444aee8eda" member_type: NON_VOTER last_known_addr { host: "127.9.155.124" port: 46355 } attrs { promote: true } } }
I20260620 19:10:26.233505 16554 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c20a27597512484cb7546601dada9ca9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c20a27597512484cb7546601dada9ca9 (127.9.155.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6d022764bb4a442e8466cb260239c7e5" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 44831 } } peers { permanent_uuid: "c20a27597512484cb7546601dada9ca9" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 35509 } attrs { promote: false } } peers { permanent_uuid: "12c2fb5fc9eb4fe2850ab1444aee8eda" member_type: NON_VOTER last_known_addr { host: "127.9.155.124" port: 46355 } attrs { promote: true } } }
I20260620 19:10:26.234537 16503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d022764bb4a442e8466cb260239c7e5 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6d022764bb4a442e8466cb260239c7e5" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6d022764bb4a442e8466cb260239c7e5" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 44831 } } peers { permanent_uuid: "c20a27597512484cb7546601dada9ca9" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 35509 } attrs { promote: false } } peers { permanent_uuid: "12c2fb5fc9eb4fe2850ab1444aee8eda" member_type: NON_VOTER last_known_addr { host: "127.9.155.124" port: 46355 } attrs { promote: true } } }
I20260620 19:10:26.234647 16503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d022764bb4a442e8466cb260239c7e5 [sys.catalog]: This master's current role is: LEADER
I20260620 19:10:26.234809 16503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d022764bb4a442e8466cb260239c7e5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6d022764bb4a442e8466cb260239c7e5" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6d022764bb4a442e8466cb260239c7e5" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 44831 } } peers { permanent_uuid: "c20a27597512484cb7546601dada9ca9" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 35509 } attrs { promote: false } } peers { permanent_uuid: "12c2fb5fc9eb4fe2850ab1444aee8eda" member_type: NON_VOTER last_known_addr { host: "127.9.155.124" port: 46355 } attrs { promote: true } } }
I20260620 19:10:26.234889 16503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d022764bb4a442e8466cb260239c7e5 [sys.catalog]: This master's current role is: LEADER
I20260620 19:10:26.235589 16691 sys_catalog.cc:455] T 00000000000000000000000000000000 P c20a27597512484cb7546601dada9ca9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6d022764bb4a442e8466cb260239c7e5" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6d022764bb4a442e8466cb260239c7e5" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 44831 } } peers { permanent_uuid: "c20a27597512484cb7546601dada9ca9" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 35509 } attrs { promote: false } } peers { permanent_uuid: "12c2fb5fc9eb4fe2850ab1444aee8eda" member_type: NON_VOTER last_known_addr { host: "127.9.155.124" port: 46355 } attrs { promote: true } } }
I20260620 19:10:26.235692 16691 sys_catalog.cc:458] T 00000000000000000000000000000000 P c20a27597512484cb7546601dada9ca9 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:10:26.239202 16516 master.cc:552] Master@127.9.155.124:46355 shutdown complete.
I20260620 19:10:26.239265 16516 master_runner.cc:305] Clearing existing system tablet
I20260620 19:10:26.239493 16516 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:26.239547 16516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:26.241593 16516 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 19:10:26.242470 16695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:26.242667 16516 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:26.242792 16516 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:10:26.242831 16516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/master-2/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/master-2/wal
uuid: "12c2fb5fc9eb4fe2850ab1444aee8eda"
format_stamp: "Formatted at 2026-06-20 19:10:26 on dist-test-slave-n3fx"
server_key: "821ad6706390d18ebed33b989f78dc2e"
server_key_iv: "ebd6887241e36b40a11f0c016b5a10d3"
server_key_version: "encryptionkey@0"
I20260620 19:10:26.244874 16516 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 12c2fb5fc9eb4fe2850ab1444aee8eda: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:10:26.247273 16516 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 12c2fb5fc9eb4fe2850ab1444aee8eda: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 19:10:26.247365 16516 log.cc:1196] T 00000000000000000000000000000000 P 12c2fb5fc9eb4fe2850ab1444aee8eda: Deleting WAL directory at /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260620 19:10:26.247653 16516 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 12c2fb5fc9eb4fe2850ab1444aee8eda: Deleting consensus metadata
I20260620 19:10:26.247821 16516 master_runner.cc:315] Copying system tablet from 127.9.155.126:44831
I20260620 19:10:26.249055 16516 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 12c2fb5fc9eb4fe2850ab1444aee8eda: tablet copy: Beginning tablet copy session from remote peer at address 127.9.155.126:44831
I20260620 19:10:26.262995 15982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:26.254356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52022 (local address 127.9.155.126:44831)
0620 19:10:26.254495 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:26.254499 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:26.254676 (+ 177us) server_negotiation.cc:407] Connection header received
0620 19:10:26.254809 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:26.254812 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:26.254863 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:26.254963 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:26.255485 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.256325 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.257546 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.257876 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.257994 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:26.259400 (+ 1406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:26.259405 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:26.259408 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:26.259439 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:26.261198 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:26.261678 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:26.261682 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:26.261684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:26.261772 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:26.262032 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:26.262035 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:26.262037 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:26.262255 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:26.262349 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:26.262700 (+ 351us) server_negotiation.cc:299] Negotiation successful
0620 19:10:26.262821 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 19:10:26.263535 16701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:26.254249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:44831 (local address 127.0.0.1:52022)
0620 19:10:26.254586 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:26.254600 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:26.254718 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:26.254997 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:26.255000 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:26.255232 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:26.255362 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:26.255370 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.256489 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.256493 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:26.257396 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:26.257422 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.258016 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.258020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:26.258125 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:26.258794 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:26.258808 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:26.259244 (+ 436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:26.261313 (+ 2069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:26.261317 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:26.261319 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:26.261577 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:26.261852 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:26.261855 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:26.261857 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:26.261948 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:26.262343 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:26.262345 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:26.262535 (+ 190us) client_negotiation.cc:769] Sending connection context
0620 19:10:26.262640 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 19:10:26.262793 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":100,"threads_started":1}
I20260620 19:10:26.264003 15963 tablet_copy_service.cc:140] P 6d022764bb4a442e8466cb260239c7e5: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 12c2fb5fc9eb4fe2850ab1444aee8eda ({username='oryx', principal='oryx/127.9.155.124@KRBTEST.COM'} at 127.0.0.1:52022)
I20260620 19:10:26.264112 15963 tablet_copy_service.cc:161] P 6d022764bb4a442e8466cb260239c7e5: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 12c2fb5fc9eb4fe2850ab1444aee8eda at {username='oryx', principal='oryx/127.9.155.124@KRBTEST.COM'} at 127.0.0.1:52022: session id = 12c2fb5fc9eb4fe2850ab1444aee8eda-00000000000000000000000000000000
I20260620 19:10:26.265228 15963 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 6d022764bb4a442e8466cb260239c7e5: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 19:10:26.266580 16516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 19:10:26.269110 16516 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 12c2fb5fc9eb4fe2850ab1444aee8eda: tablet copy: Starting download of 0 data blocks...
I20260620 19:10:26.269389 16516 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 12c2fb5fc9eb4fe2850ab1444aee8eda: tablet copy: Starting download of 1 WAL segments...
I20260620 19:10:26.270905 16516 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 12c2fb5fc9eb4fe2850ab1444aee8eda: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 19:10:26.272830 15963 tablet_copy_service.cc:342] P 6d022764bb4a442e8466cb260239c7e5: Request end of tablet copy session 12c2fb5fc9eb4fe2850ab1444aee8eda-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.9.155.124@KRBTEST.COM'} at 127.0.0.1:52022
I20260620 19:10:26.272926 15963 tablet_copy_service.cc:433] P 6d022764bb4a442e8466cb260239c7e5: ending tablet copy session 12c2fb5fc9eb4fe2850ab1444aee8eda-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 12c2fb5fc9eb4fe2850ab1444aee8eda
I20260620 19:10:26.280442 16516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:10:26.282294 16705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:26.282509 16703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:26.282498 16702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:26.282716 16516 server_base.cc:1034] running on GCE node
I20260620 19:10:26.282860 16516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:26.283167 16516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:26.284286 16516 hybrid_clock.cc:648] HybridClock initialized: now 1781982626284275 us; error 32 us; skew 500 ppm
Jun 20 19:10:26 dist-test-slave-n3fx krb5kdc[15898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982626, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.155.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:26.285816 16516 init.cc:377] Logged in from keytab as oryx/127.9.155.124@KRBTEST.COM (short username oryx)
I20260620 19:10:26.286355 16516 webserver.cc:460] Webserver started at http://127.9.155.124:43701/ using document root <none> and password file <none>
I20260620 19:10:26.286540 16516 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:26.286583 16516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:26.288057 16516 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:10:26.289069 16712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:26.289300 16516 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:10:26.289377 16516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/master-2/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/master-2/wal
uuid: "12c2fb5fc9eb4fe2850ab1444aee8eda"
format_stamp: "Formatted at 2026-06-20 19:10:26 on dist-test-slave-n3fx"
server_key: "821ad6706390d18ebed33b989f78dc2e"
server_key_iv: "ebd6887241e36b40a11f0c016b5a10d3"
server_key_version: "encryptionkey@0"
I20260620 19:10:26.289464 16516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:26.304078 16516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:26.306373 16516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:26.311199 16516 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.124:46355
I20260620 19:10:26.311229 16764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.124:46355 every 8 connection(s)
I20260620 19:10:26.311770 16516 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/master-2/data/info.pb
I20260620 19:10:26.313586 16765 sys_catalog.cc:263] Verifying existing consensus state
I20260620 19:10:26.314682 16765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12c2fb5fc9eb4fe2850ab1444aee8eda: Bootstrap starting.
I20260620 19:10:26.329934 16765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12c2fb5fc9eb4fe2850ab1444aee8eda: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:10:26.330175 16765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12c2fb5fc9eb4fe2850ab1444aee8eda: Bootstrap complete.
I20260620 19:10:26.336807 16765 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12c2fb5fc9eb4fe2850ab1444aee8eda [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6d022764bb4a442e8466cb260239c7e5" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 44831 } } peers { permanent_uuid: "c20a27597512484cb7546601dada9ca9" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 35509 } attrs { promote: false } } peers { permanent_uuid: "12c2fb5fc9eb4fe2850ab1444aee8eda" member_type: NON_VOTER last_known_addr { host: "127.9.155.124" port: 46355 } attrs { promote: true } }
I20260620 19:10:26.337018 16765 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12c2fb5fc9eb4fe2850ab1444aee8eda [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 12c2fb5fc9eb4fe2850ab1444aee8eda, State: Initialized, Role: LEARNER
I20260620 19:10:26.337142 16765 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12c2fb5fc9eb4fe2850ab1444aee8eda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6d022764bb4a442e8466cb260239c7e5" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 44831 } } peers { permanent_uuid: "c20a27597512484cb7546601dada9ca9" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 35509 } attrs { promote: false } } peers { permanent_uuid: "12c2fb5fc9eb4fe2850ab1444aee8eda" member_type: NON_VOTER last_known_addr { host: "127.9.155.124" port: 46355 } attrs { promote: true } }
I20260620 19:10:26.337637 16765 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12c2fb5fc9eb4fe2850ab1444aee8eda [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:10:26.342760 16779 catalog_manager.cc:1259] Loaded cluster ID: 3e1c5ad4f96242d5b1943107518facd5
I20260620 19:10:26.342821 16779 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 12c2fb5fc9eb4fe2850ab1444aee8eda: loading cluster ID for follower catalog manager: success
I20260620 19:10:26.341406 16768 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12c2fb5fc9eb4fe2850ab1444aee8eda [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6d022764bb4a442e8466cb260239c7e5" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 44831 } } peers { permanent_uuid: "c20a27597512484cb7546601dada9ca9" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 35509 } attrs { promote: false } } peers { permanent_uuid: "12c2fb5fc9eb4fe2850ab1444aee8eda" member_type: NON_VOTER last_known_addr { host: "127.9.155.124" port: 46355 } attrs { promote: true } } }
I20260620 19:10:26.344493 16768 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12c2fb5fc9eb4fe2850ab1444aee8eda [sys.catalog]: This master's current role is: LEARNER
I20260620 19:10:26.344969 16779 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 12c2fb5fc9eb4fe2850ab1444aee8eda: acquiring CA information for follower catalog manager: success
I20260620 19:10:26.346056 16779 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 12c2fb5fc9eb4fe2850ab1444aee8eda: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 19:10:26.350632 16688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:26.350999 16688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:26.351064 16688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 19:10:26.352393 16268 heartbeater.cc:503] Master 127.9.155.126:44831 was elected leader, sending a full tablet report...
W20260620 19:10:26.357100 16688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:26.357183 16688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:26.357270 16688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.65
I20260620 19:10:26.363073 16688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:43463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/krb5kdc/oryx_127.9.155.65.keytab
--principal=oryx/127.9.155.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.65:42443
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.155.65
--webserver_port=37505
--tserver_master_addrs=127.9.155.126:44831,127.9.155.125:35509,127.9.155.124:46355
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16688
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.65
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:10:26.364105 16688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:26.365126 16688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:10:26.374923 16785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:26.375036 16784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:26.375123 16688 server_base.cc:1034] running on GCE node
W20260620 19:10:26.374948 16787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:26.375697 16688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:26.376427 16688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:26.377606 16688 hybrid_clock.cc:648] HybridClock initialized: now 1781982626377594 us; error 28 us; skew 500 ppm
Jun 20 19:10:26 dist-test-slave-n3fx krb5kdc[15898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982626, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.155.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:26.380435 16688 init.cc:377] Logged in from keytab as oryx/127.9.155.65@KRBTEST.COM (short username oryx)
I20260620 19:10:26.381333 16688 webserver.cc:460] Webserver started at http://127.9.155.65:37505/ using document root <none> and password file <none>
I20260620 19:10:26.381848 16688 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:26.381917 16688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:26.385883 16688 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 19:10:26.388080 16794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:26.389545 16688 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 19:10:26.389653 16688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-0/wal
uuid: "af15d4c4a9304cc8a20af15051256305"
format_stamp: "Formatted at 2026-06-20 19:10:25 on dist-test-slave-n3fx"
server_key: "84ebcd940b8416f96e97ca4c5fe018e0"
server_key_iv: "ba1ba47c27c3b058fdead948bcf2b0fc"
server_key_version: "encryptionkey@0"
I20260620 19:10:26.390277 16688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:26.411450 16688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:26.414367 16688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:26.414535 16688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:26.415143 16688 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:10:26.416023 16688 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:10:26.416070 16688 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:26.416108 16688 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:10:26.416122 16688 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:26.424826 16688 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.65:42443
I20260620 19:10:26.424861 16907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.65:42443 every 8 connection(s)
I20260620 19:10:26.425781 16688 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
I20260620 19:10:26.428973 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 16688
I20260620 19:10:26.429154 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 16139
I20260620 19:10:26.436106 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.66:44711
--local_ip_for_outbound_sockets=127.9.155.66
--tserver_master_addrs=127.9.155.126:44831
--webserver_port=35877
--webserver_interface=127.9.155.66
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/krb5kdc/oryx_127.9.155.66.keytab
--principal=oryx/127.9.155.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:43463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.9.155.126:44831,127.9.155.125:35509
--tserver_master_addrs=127.9.155.126:44831,127.9.155.125:35509,127.9.155.124:46355 with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 19:10:26 dist-test-slave-n3fx krb5kdc[15898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982626, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.155.65@KRBTEST.COM for oryx/127.9.155.125@KRBTEST.COM
Jun 20 19:10:26 dist-test-slave-n3fx krb5kdc[15898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982626, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.155.65@KRBTEST.COM for oryx/127.9.155.124@KRBTEST.COM
Jun 20 19:10:26 dist-test-slave-n3fx krb5kdc[15898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982626, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.155.65@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM
I20260620 19:10:26.444006 16666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:26.427755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.65:49525 (local address 127.9.155.125:35509)
0620 19:10:26.427854 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:26.427857 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:26.429729 (+ 1872us) server_negotiation.cc:407] Connection header received
0620 19:10:26.430573 (+ 844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:26.430577 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:26.430633 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:26.430734 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:26.432811 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.433613 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.435204 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.435543 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.435619 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:26.439557 (+ 3938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:26.439560 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:26.439563 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:26.439589 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:26.441509 (+ 1920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:26.442207 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:26.442210 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:26.442212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:26.442295 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:26.442667 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:26.442670 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:26.442672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:26.442909 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:26.442987 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:26.443709 (+ 722us) server_negotiation.cc:299] Negotiation successful
0620 19:10:26.443830 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 19:10:26.444010 15982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:26.427603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.65:34159 (local address 127.9.155.126:44831)
0620 19:10:26.427736 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:26.427740 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:26.429519 (+ 1779us) server_negotiation.cc:407] Connection header received
0620 19:10:26.431014 (+ 1495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:26.431017 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:26.431070 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:26.431148 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:26.432784 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.433607 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.436060 (+ 2453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.436393 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.436469 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:26.439690 (+ 3221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:26.439693 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:26.439696 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:26.439724 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:26.441625 (+ 1901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:26.442240 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:26.442243 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:26.442245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:26.442405 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:26.442738 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:26.442742 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:26.442744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:26.442989 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:26.443066 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:26.443726 (+ 660us) server_negotiation.cc:299] Negotiation successful
0620 19:10:26.443851 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 19:10:26.444988 16915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:26.428573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.65:60947 (local address 127.9.155.124:46355)
0620 19:10:26.431549 (+ 2976us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:26.431554 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:26.431568 (+ 14us) server_negotiation.cc:407] Connection header received
0620 19:10:26.431608 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:26.431611 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:26.431659 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:26.431740 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:26.432820 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.433846 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.435326 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.435643 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.435728 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:26.439518 (+ 3790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:26.439522 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:26.439525 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:26.439552 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:26.441957 (+ 2405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:26.442417 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:26.442420 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:26.442422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:26.442499 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:26.442775 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:26.442778 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:26.442780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:26.442970 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:26.443045 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:26.444710 (+ 1665us) server_negotiation.cc:299] Negotiation successful
0620 19:10:26.444839 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2893,"thread_start_us":75,"threads_started":1}
I20260620 19:10:26.445408 16914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:26.428141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:44831 (local address 127.9.155.65:34159)
0620 19:10:26.429376 (+ 1235us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:26.429427 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:26.430909 (+ 1482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:26.431266 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:26.431270 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:26.431813 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:26.432617 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:26.432638 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.433774 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.433777 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:26.435929 (+ 2152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:26.435940 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.436639 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.436642 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:26.436746 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:26.437216 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:26.437230 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:26.439554 (+ 2324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:26.441758 (+ 2204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:26.441761 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:26.441763 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:26.442119 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:26.442525 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:26.442528 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:26.442530 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:26.442619 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:26.443142 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:26.443145 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:26.443427 (+ 282us) client_negotiation.cc:769] Sending connection context
0620 19:10:26.444164 (+ 737us) client_negotiation.cc:240] Negotiation successful
0620 19:10:26.444400 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1027,"mutex_wait_us":1275,"spinlock_wait_cycles":30464,"thread_start_us":137,"threads_started":1}
I20260620 19:10:26.445638 16913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:26.428141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.125:35509 (local address 127.9.155.65:49525)
0620 19:10:26.429645 (+ 1504us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:26.429655 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:26.430384 (+ 729us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:26.431222 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:26.431231 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:26.431824 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:26.432618 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:26.432638 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.433775 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.433778 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:26.435068 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:26.435078 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.435831 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.435835 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:26.436097 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:26.436937 (+ 840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:26.436965 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:26.439410 (+ 2445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:26.441644 (+ 2234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:26.441650 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:26.441667 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:26.442093 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:26.442398 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:26.442401 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:26.442403 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:26.442572 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:26.443053 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:26.443059 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:26.443425 (+ 366us) client_negotiation.cc:769] Sending connection context
0620 19:10:26.444161 (+ 736us) client_negotiation.cc:240] Negotiation successful
0620 19:10:26.444327 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1426,"mutex_wait_us":346,"spinlock_wait_cycles":60160,"thread_start_us":148,"threads_started":1}
I20260620 19:10:26.446017 16912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:26.428407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.124:46355 (local address 127.9.155.65:60947)
0620 19:10:26.430719 (+ 2312us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:26.430731 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:26.430814 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:26.431861 (+ 1047us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:26.431864 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:26.432016 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:26.432699 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:26.432706 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.433957 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.433961 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:26.435175 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:26.435185 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.435831 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.435835 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:26.436093 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:26.436937 (+ 844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:26.436965 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:26.439339 (+ 2374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:26.442079 (+ 2740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:26.442082 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:26.442084 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:26.442307 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:26.442604 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:26.442607 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:26.442609 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:26.442685 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:26.443079 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:26.443081 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:26.443425 (+ 344us) client_negotiation.cc:769] Sending connection context
0620 19:10:26.443642 (+ 217us) client_negotiation.cc:240] Negotiation successful
0620 19:10:26.443970 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2254,"mutex_wait_us":727,"spinlock_wait_cycles":19072,"thread_start_us":80,"threads_started":1}
I20260620 19:10:26.446763 16908 heartbeater.cc:347] Connected to a master server at 127.9.155.124:46355
I20260620 19:10:26.447039 16908 heartbeater.cc:464] Registering TS with master...
I20260620 19:10:26.447670 16908 heartbeater.cc:511] Master 127.9.155.124:46355 requested a full tablet report, sending...
I20260620 19:10:26.448254 16909 heartbeater.cc:347] Connected to a master server at 127.9.155.126:44831
I20260620 19:10:26.448338 16909 heartbeater.cc:464] Registering TS with master...
I20260620 19:10:26.448514 16909 heartbeater.cc:511] Master 127.9.155.126:44831 requested a full tablet report, sending...
I20260620 19:10:26.448866 16910 heartbeater.cc:347] Connected to a master server at 127.9.155.125:35509
I20260620 19:10:26.448938 16910 heartbeater.cc:464] Registering TS with master...
I20260620 19:10:26.449074 16910 heartbeater.cc:511] Master 127.9.155.125:35509 requested a full tablet report, sending...
I20260620 19:10:26.449322 15942 ts_manager.cc:194] Re-registered known tserver with Master: af15d4c4a9304cc8a20af15051256305 (127.9.155.65:42443)
I20260620 19:10:26.449550 16729 ts_manager.cc:194] Registered new tserver with Master: af15d4c4a9304cc8a20af15051256305 (127.9.155.65:42443)
I20260620 19:10:26.450079 15942 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.155.65@KRBTEST.COM'} at 127.9.155.65:34159
I20260620 19:10:26.450114 16544 ts_manager.cc:194] Registered new tserver with Master: af15d4c4a9304cc8a20af15051256305 (127.9.155.65:42443)
W20260620 19:10:26.553319 16916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:26.553591 16916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:26.553635 16916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:26.558768 16916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:26.558858 16916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:26.558947 16916 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.66
I20260620 19:10:26.564286 16916 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:43463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/krb5kdc/oryx_127.9.155.66.keytab
--principal=oryx/127.9.155.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.66:44711
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.155.66
--webserver_port=35877
--tserver_master_addrs=127.9.155.126:44831,127.9.155.125:35509,127.9.155.124:46355
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16916
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.66
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:10:26.565284 16916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:26.566228 16916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:10:26.575758 16922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:26.575759 16921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:26.575964 16916 server_base.cc:1034] running on GCE node
W20260620 19:10:26.575758 16924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:26.576399 16402 heartbeater.cc:503] Master 127.9.155.126:44831 was elected leader, sending a full tablet report...
I20260620 19:10:26.576648 16916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:26.577389 16916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:26.578559 16916 hybrid_clock.cc:648] HybridClock initialized: now 1781982626578536 us; error 40 us; skew 500 ppm
Jun 20 19:10:26 dist-test-slave-n3fx krb5kdc[15898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982626, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.155.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:26.581357 16916 init.cc:377] Logged in from keytab as oryx/127.9.155.66@KRBTEST.COM (short username oryx)
I20260620 19:10:26.582267 16916 webserver.cc:460] Webserver started at http://127.9.155.66:35877/ using document root <none> and password file <none>
I20260620 19:10:26.582844 16916 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:26.582950 16916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:26.586931 16916 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 19:10:26.589186 16931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:26.590466 16916 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 19:10:26.590582 16916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-1/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-1/wal
uuid: "22fadda898e54ffd9a8eaac7a55fcb91"
format_stamp: "Formatted at 2026-06-20 19:10:25 on dist-test-slave-n3fx"
server_key: "c67010f8c0c07fee9aa50373b76b10e6"
server_key_iv: "8879281e625aaadead4c2cfd17490eff"
server_key_version: "encryptionkey@0"
I20260620 19:10:26.591125 16916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:26.603116 16916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:26.606287 16916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:26.606467 16916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:26.607050 16916 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:10:26.607940 16916 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:10:26.607990 16916 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:26.608042 16916 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:10:26.608070 16916 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:26.616537 16916 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.66:44711
I20260620 19:10:26.616564 17044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.66:44711 every 8 connection(s)
I20260620 19:10:26.617547 16916 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
I20260620 19:10:26.622677 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 16916
I20260620 19:10:26.622900 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 16273
Jun 20 19:10:26 dist-test-slave-n3fx krb5kdc[15898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982626, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.155.66@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM
Jun 20 19:10:26 dist-test-slave-n3fx krb5kdc[15898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982626, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.155.66@KRBTEST.COM for oryx/127.9.155.124@KRBTEST.COM
I20260620 19:10:26.630366 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.67:34277
--local_ip_for_outbound_sockets=127.9.155.67
--tserver_master_addrs=127.9.155.126:44831
--webserver_port=39105
--webserver_interface=127.9.155.67
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/krb5kdc/oryx_127.9.155.67.keytab
--principal=oryx/127.9.155.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:43463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.9.155.126:44831,127.9.155.125:35509
--tserver_master_addrs=127.9.155.126:44831,127.9.155.125:35509,127.9.155.124:46355 with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 19:10:26 dist-test-slave-n3fx krb5kdc[15898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982626, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.155.66@KRBTEST.COM for oryx/127.9.155.125@KRBTEST.COM
I20260620 19:10:26.634904 16915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:26.619627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.66:58735 (local address 127.9.155.124:46355)
0620 19:10:26.619751 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:26.619755 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:26.620530 (+ 775us) server_negotiation.cc:407] Connection header received
0620 19:10:26.621704 (+ 1174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:26.621708 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:26.621778 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:26.621856 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:26.624306 (+ 2450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.625120 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.626300 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.626585 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.626681 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:26.630028 (+ 3347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:26.630032 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:26.630035 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:26.630066 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:26.631968 (+ 1902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:26.632808 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:26.632812 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:26.632814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:26.632914 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:26.633468 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:26.633472 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:26.633474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:26.633698 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:26.633796 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:26.634569 (+ 773us) server_negotiation.cc:299] Negotiation successful
0620 19:10:26.634718 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 19:10:26.634904 15982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:26.619384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.66:42103 (local address 127.9.155.126:44831)
0620 19:10:26.619528 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:26.619532 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:26.621216 (+ 1684us) server_negotiation.cc:407] Connection header received
0620 19:10:26.621728 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:26.621732 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:26.621785 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:26.621834 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:26.624134 (+ 2300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.624973 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.626503 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.626806 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.626905 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:26.629637 (+ 2732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:26.629642 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:26.629645 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:26.629679 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:26.632002 (+ 2323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:26.632802 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:26.632806 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:26.632808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:26.633016 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:26.633545 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:26.633548 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:26.633550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:26.633748 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:26.633821 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:26.634643 (+ 822us) server_negotiation.cc:299] Negotiation successful
0620 19:10:26.634756 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 19:10:26.636652 17051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:26.619850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:44831 (local address 127.9.155.66:42103)
0620 19:10:26.621107 (+ 1257us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:26.621119 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:26.621489 (+ 370us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:26.622070 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:26.622077 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:26.622730 (+ 653us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:26.623970 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:26.623988 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.625131 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.625135 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:26.626365 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:26.626377 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.626911 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.626915 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:26.626999 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:26.627624 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:26.627645 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:26.629414 (+ 1769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:26.632119 (+ 2705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:26.632127 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:26.632149 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:26.632677 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:26.633345 (+ 668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:26.633349 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:26.633351 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:26.633445 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:26.633877 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:26.633879 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:26.634259 (+ 380us) client_negotiation.cc:769] Sending connection context
0620 19:10:26.635720 (+ 1461us) client_negotiation.cc:240] Negotiation successful
0620 19:10:26.635872 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1178,"mutex_wait_us":239,"spinlock_wait_cycles":26112,"thread_start_us":125,"threads_started":1}
I20260620 19:10:26.636940 17049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:26.619850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.124:46355 (local address 127.9.155.66:58735)
0620 19:10:26.620376 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:26.620424 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:26.621465 (+ 1041us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:26.622071 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:26.622080 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:26.622729 (+ 649us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:26.623972 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:26.623989 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.625216 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.625220 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:26.626171 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:26.626181 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.626734 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.626738 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:26.626967 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:26.627682 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:26.627696 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:26.629840 (+ 2144us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:26.632119 (+ 2279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:26.632127 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:26.632149 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:26.632677 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:26.633148 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:26.633151 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:26.633153 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:26.633358 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:26.633867 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:26.633874 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:26.634259 (+ 385us) client_negotiation.cc:769] Sending connection context
0620 19:10:26.635131 (+ 872us) client_negotiation.cc:240] Negotiation successful
0620 19:10:26.635522 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"mutex_wait_us":131,"spinlock_wait_cycles":24576,"thread_start_us":125,"threads_started":1}
I20260620 19:10:26.638175 17045 heartbeater.cc:347] Connected to a master server at 127.9.155.124:46355
I20260620 19:10:26.638433 17045 heartbeater.cc:464] Registering TS with master...
I20260620 19:10:26.638671 16666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:26.619865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.66:48961 (local address 127.9.155.125:35509)
0620 19:10:26.619998 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:26.620002 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:26.620995 (+ 993us) server_negotiation.cc:407] Connection header received
0620 19:10:26.621797 (+ 802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:26.621800 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:26.621854 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:26.621930 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:26.630149 (+ 8219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.630918 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.632141 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.632532 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.632612 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:26.634991 (+ 2379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:26.634995 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:26.634998 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:26.635030 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:26.636696 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:26.637251 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:26.637254 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:26.637256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:26.637348 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:26.637674 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:26.637677 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:26.637679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:26.637901 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:26.637975 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:26.638386 (+ 411us) server_negotiation.cc:299] Negotiation successful
0620 19:10:26.638521 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 19:10:26.639016 17045 heartbeater.cc:511] Master 127.9.155.124:46355 requested a full tablet report, sending...
I20260620 19:10:26.639189 17050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:26.619850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.125:35509 (local address 127.9.155.66:48961)
0620 19:10:26.620880 (+ 1030us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:26.620893 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:26.621473 (+ 580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:26.622071 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:26.622080 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:26.622729 (+ 649us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:26.629855 (+ 7126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:26.629923 (+ 68us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.631077 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.631081 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:26.631992 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:26.632004 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.632790 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.632794 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:26.632895 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:26.633349 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:26.633374 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:26.634814 (+ 1440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:26.636823 (+ 2009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:26.636827 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:26.636828 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:26.637086 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:26.637477 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:26.637480 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:26.637482 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:26.637569 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:26.638048 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:26.638051 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:26.638143 (+ 92us) client_negotiation.cc:769] Sending connection context
0620 19:10:26.638818 (+ 675us) client_negotiation.cc:240] Negotiation successful
0620 19:10:26.638964 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":948,"mutex_wait_us":40,"spinlock_wait_cycles":19712,"thread_start_us":149,"threads_started":1}
I20260620 19:10:26.638154 17046 heartbeater.cc:347] Connected to a master server at 127.9.155.126:44831
I20260620 19:10:26.639595 17046 heartbeater.cc:464] Registering TS with master...
I20260620 19:10:26.639770 17046 heartbeater.cc:511] Master 127.9.155.126:44831 requested a full tablet report, sending...
I20260620 19:10:26.639846 17047 heartbeater.cc:347] Connected to a master server at 127.9.155.125:35509
I20260620 19:10:26.639904 17047 heartbeater.cc:464] Registering TS with master...
I20260620 19:10:26.640038 17047 heartbeater.cc:511] Master 127.9.155.125:35509 requested a full tablet report, sending...
I20260620 19:10:26.640576 15942 ts_manager.cc:194] Re-registered known tserver with Master: 22fadda898e54ffd9a8eaac7a55fcb91 (127.9.155.66:44711)
I20260620 19:10:26.640647 16729 ts_manager.cc:194] Registered new tserver with Master: 22fadda898e54ffd9a8eaac7a55fcb91 (127.9.155.66:44711)
I20260620 19:10:26.640908 16544 ts_manager.cc:194] Registered new tserver with Master: 22fadda898e54ffd9a8eaac7a55fcb91 (127.9.155.66:44711)
I20260620 19:10:26.641312 15942 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.155.66@KRBTEST.COM'} at 127.9.155.66:42103
W20260620 19:10:26.746563 17052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:26.746825 17052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:26.746868 17052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:26.751118 17052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:26.751191 17052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:26.751251 17052 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.67
I20260620 19:10:26.756584 17052 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:43463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/krb5kdc/oryx_127.9.155.67.keytab
--principal=oryx/127.9.155.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.67:34277
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.155.67
--webserver_port=39105
--tserver_master_addrs=127.9.155.126:44831,127.9.155.125:35509,127.9.155.124:46355
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17052
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.67
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:10:26.757628 17052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:26.758601 17052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 19:10:26.763571 17057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:26.759707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.124:46355 (local address 127.0.0.1:42958)
0620 19:10:26.759915 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:26.759930 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:26.760029 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:26.760364 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:26.760366 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:26.760380 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:10:26.760503 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:26.760510 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.761569 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.761572 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:26.762334 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:26.762344 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.763215 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.763217 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:26.763290 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:26.763311 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 19:10:26.763353 (+ 42us) client_negotiation.cc:240] Negotiation successful
0620 19:10:26.763420 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":55,"threads_started":1}
I20260620 19:10:26.763567 16915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:26.759834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42958 (local address 127.9.155.124:46355)
0620 19:10:26.759972 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:26.759976 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:26.759996 (+ 20us) server_negotiation.cc:407] Connection header received
0620 19:10:26.760158 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:26.760162 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:26.760253 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:26.760355 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:10:26.760612 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.761418 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.762474 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.763093 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.763197 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:26.763347 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:26.763381 (+ 34us) server_negotiation.cc:299] Negotiation successful
0620 19:10:26.763447 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 19:10:26.764314 16739 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 12c2fb5fc9eb4fe2850ab1444aee8eda [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260620 19:10:26.764709 16768 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12c2fb5fc9eb4fe2850ab1444aee8eda [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d022764bb4a442e8466cb260239c7e5. Latest consensus state: current_term: 1 leader_uuid: "6d022764bb4a442e8466cb260239c7e5" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6d022764bb4a442e8466cb260239c7e5" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 44831 } } peers { permanent_uuid: "c20a27597512484cb7546601dada9ca9" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 35509 } attrs { promote: false } } peers { permanent_uuid: "12c2fb5fc9eb4fe2850ab1444aee8eda" member_type: NON_VOTER last_known_addr { host: "127.9.155.124" port: 46355 } attrs { promote: true } } }
I20260620 19:10:26.764849 16768 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12c2fb5fc9eb4fe2850ab1444aee8eda [sys.catalog]: This master's current role is: LEARNER
I20260620 19:10:26.769526 17052 server_base.cc:1034] running on GCE node
W20260620 19:10:26.769410 17059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:26.769418 17062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:26.769414 17060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:26.770063 17052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:26.770707 17052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:26.771860 17052 hybrid_clock.cc:648] HybridClock initialized: now 1781982626771822 us; error 49 us; skew 500 ppm
Jun 20 19:10:26 dist-test-slave-n3fx krb5kdc[15898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982626, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.155.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:26.774580 17052 init.cc:377] Logged in from keytab as oryx/127.9.155.67@KRBTEST.COM (short username oryx)
I20260620 19:10:26.775420 17052 webserver.cc:460] Webserver started at http://127.9.155.67:39105/ using document root <none> and password file <none>
I20260620 19:10:26.775946 17052 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:26.776016 17052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:26.779829 17052 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 19:10:26.781935 17069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:26.783267 17052 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 19:10:26.783449 17052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-2/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-2/wal
uuid: "a2b48205cf584ab18dd060cf238791ab"
format_stamp: "Formatted at 2026-06-20 19:10:25 on dist-test-slave-n3fx"
server_key: "366067e8aeb3ef19a1645e8e27abaedd"
server_key_iv: "e82af44c38030237cfb9a09315afdee8"
server_key_version: "encryptionkey@0"
I20260620 19:10:26.784015 17052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:26.795562 17052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:26.798807 17052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:26.798992 17052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:26.799588 17052 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:10:26.800463 17052 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:10:26.800513 17052 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:26.800550 17052 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:10:26.800578 17052 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:26.809368 17052 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.67:34277
I20260620 19:10:26.809377 17182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.67:34277 every 8 connection(s)
I20260620 19:10:26.810438 17052 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
I20260620 19:10:26.817586 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 17052
Jun 20 19:10:26 dist-test-slave-n3fx krb5kdc[15898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982626, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.155.67@KRBTEST.COM for oryx/127.9.155.124@KRBTEST.COM
Jun 20 19:10:26 dist-test-slave-n3fx krb5kdc[15898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982626, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.155.67@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM
Jun 20 19:10:26 dist-test-slave-n3fx krb5kdc[15898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982626, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.155.67@KRBTEST.COM for oryx/127.9.155.125@KRBTEST.COM
I20260620 19:10:26.828305 16666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:26.812332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.67:60617 (local address 127.9.155.125:35509)
0620 19:10:26.812432 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:26.812436 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:26.814140 (+ 1704us) server_negotiation.cc:407] Connection header received
0620 19:10:26.814596 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:26.814599 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:26.814651 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:26.814706 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:26.816177 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.816747 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.818751 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.819099 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.819175 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:26.823445 (+ 4270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:26.823449 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:26.823452 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:26.823486 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:26.825414 (+ 1928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:26.826722 (+ 1308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:26.826727 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:26.826729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:26.826814 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:26.827131 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:26.827134 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:26.827136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:26.827353 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:26.827435 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:26.827978 (+ 543us) server_negotiation.cc:299] Negotiation successful
0620 19:10:26.828116 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 19:10:26.829233 16915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:26.812782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.67:54979 (local address 127.9.155.124:46355)
0620 19:10:26.812894 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:26.812898 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:26.813443 (+ 545us) server_negotiation.cc:407] Connection header received
0620 19:10:26.814592 (+ 1149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:26.814596 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:26.814662 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:26.814738 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:26.816964 (+ 2226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.817551 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.818751 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.819066 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.819144 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:26.823796 (+ 4652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:26.823799 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:26.823802 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:26.823828 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:26.825676 (+ 1848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:26.826115 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:26.826118 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:26.826120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:26.826194 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:26.826837 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:26.826840 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:26.826842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:26.827043 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:26.827120 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:26.828657 (+ 1537us) server_negotiation.cc:299] Negotiation successful
0620 19:10:26.829102 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 19:10:26.829528 17187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:26.812825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.124:46355 (local address 127.9.155.67:54979)
0620 19:10:26.813298 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:26.813348 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:26.814360 (+ 1012us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:26.814934 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:26.814943 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:26.815469 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:26.816042 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:26.816049 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.817681 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.817684 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:26.818618 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:26.818628 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.819452 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.819455 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:26.819794 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:26.820671 (+ 877us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:26.820683 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:26.823679 (+ 2996us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:26.825792 (+ 2113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:26.825796 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:26.825798 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:26.826018 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:26.826547 (+ 529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:26.826550 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:26.826552 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:26.826736 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:26.827169 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:26.827176 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:26.827507 (+ 331us) client_negotiation.cc:769] Sending connection context
0620 19:10:26.827699 (+ 192us) client_negotiation.cc:240] Negotiation successful
0620 19:10:26.828062 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"mutex_wait_us":144,"spinlock_wait_cycles":68608,"thread_start_us":111,"threads_started":1}
I20260620 19:10:26.829751 17189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:26.812825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.125:35509 (local address 127.9.155.67:60617)
0620 19:10:26.814029 (+ 1204us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:26.814041 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:26.814372 (+ 331us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:26.814934 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:26.814941 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:26.815468 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:26.816029 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:26.816044 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.816882 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.816885 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:26.818620 (+ 1735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:26.818626 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.819431 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.819436 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:26.819725 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:26.820562 (+ 837us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:26.820583 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:26.823307 (+ 2724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:26.825526 (+ 2219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:26.825532 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:26.825550 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:26.825879 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:26.826921 (+ 1042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:26.826924 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:26.826926 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:26.827030 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:26.827479 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:26.827482 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:26.827577 (+ 95us) client_negotiation.cc:769] Sending connection context
0620 19:10:26.827699 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 19:10:26.828627 (+ 928us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1121,"mutex_wait_us":1036,"spinlock_wait_cycles":44672,"thread_start_us":118,"threads_started":1}
I20260620 19:10:26.831032 17183 heartbeater.cc:347] Connected to a master server at 127.9.155.124:46355
I20260620 19:10:26.831075 17185 heartbeater.cc:347] Connected to a master server at 127.9.155.125:35509
I20260620 19:10:26.831277 17185 heartbeater.cc:464] Registering TS with master...
I20260620 19:10:26.831278 17183 heartbeater.cc:464] Registering TS with master...
I20260620 19:10:26.831858 17194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:26.819565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42970 (local address 127.9.155.124:46355)
0620 19:10:26.820264 (+ 699us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:26.820276 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:26.821744 (+ 1468us) server_negotiation.cc:407] Connection header received
0620 19:10:26.821935 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:26.821938 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:26.822004 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:26.822066 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:26.823432 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.824249 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.825409 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.825733 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.825811 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:26.827329 (+ 1518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:26.827333 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:26.827335 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:26.827357 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:26.829003 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:26.829801 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:26.829805 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:26.829806 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:26.829883 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:26.830614 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:26.830618 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:26.830620 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:26.830830 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:26.830932 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:26.831589 (+ 657us) server_negotiation.cc:299] Negotiation successful
0620 19:10:26.831704 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":78,"threads_started":1}
I20260620 19:10:26.831873 17185 heartbeater.cc:511] Master 127.9.155.125:35509 requested a full tablet report, sending...
I20260620 19:10:26.832849 17183 heartbeater.cc:511] Master 127.9.155.124:46355 requested a full tablet report, sending...
I20260620 19:10:26.833101 16544 ts_manager.cc:194] Registered new tserver with Master: a2b48205cf584ab18dd060cf238791ab (127.9.155.67:34277)
I20260620 19:10:26.833523 16729 ts_manager.cc:194] Registered new tserver with Master: a2b48205cf584ab18dd060cf238791ab (127.9.155.67:34277)
I20260620 19:10:26.833626 15982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:26.812281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.67:49839 (local address 127.9.155.126:44831)
0620 19:10:26.812415 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:26.812419 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:26.813912 (+ 1493us) server_negotiation.cc:407] Connection header received
0620 19:10:26.814695 (+ 783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:26.814697 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:26.814755 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:26.814823 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:26.816308 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.816840 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.818757 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.819113 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.819176 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:26.823289 (+ 4113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:26.823293 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:26.823296 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:26.823323 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:26.826599 (+ 3276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:26.832361 (+ 5762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:26.832365 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:26.832366 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:26.832457 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:26.832753 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:26.832757 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:26.832758 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:26.832972 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:26.833049 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:26.833369 (+ 320us) server_negotiation.cc:299] Negotiation successful
0620 19:10:26.833487 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 19:10:26.834092 17188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:26.812825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:44831 (local address 127.9.155.67:49839)
0620 19:10:26.813801 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:26.813813 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:26.814362 (+ 549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:26.814934 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:26.814943 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:26.815480 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:26.816039 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:26.816049 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.816914 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.816917 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:26.818605 (+ 1688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:26.818624 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.819431 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.819436 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:26.819731 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:26.820611 (+ 880us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:26.820627 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:26.823116 (+ 2489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:26.831954 (+ 8838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:26.831957 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:26.831970 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:26.832242 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:26.832568 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:26.832571 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:26.832573 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:26.832655 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:26.833101 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:26.833104 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:26.833191 (+ 87us) client_negotiation.cc:769] Sending connection context
0620 19:10:26.833764 (+ 573us) client_negotiation.cc:240] Negotiation successful
0620 19:10:26.833905 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":893,"mutex_wait_us":408,"spinlock_wait_cycles":27776,"thread_start_us":134,"threads_started":1}
I20260620 19:10:26.836294 17184 heartbeater.cc:347] Connected to a master server at 127.9.155.126:44831
I20260620 19:10:26.836390 17184 heartbeater.cc:464] Registering TS with master...
I20260620 19:10:26.836572 17184 heartbeater.cc:511] Master 127.9.155.126:44831 requested a full tablet report, sending...
I20260620 19:10:26.837183 15942 ts_manager.cc:194] Re-registered known tserver with Master: a2b48205cf584ab18dd060cf238791ab (127.9.155.67:34277)
I20260620 19:10:26.837770 15942 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.155.67@KRBTEST.COM'} at 127.9.155.67:49839
I20260620 19:10:26.841234 17198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:26.821000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34324 (local address 127.9.155.125:35509)
0620 19:10:26.822828 (+ 1828us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:26.822833 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:26.822846 (+ 13us) server_negotiation.cc:407] Connection header received
0620 19:10:26.822882 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:26.822884 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:26.822927 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:26.831020 (+ 8093us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:26.831121 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.831839 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.835671 (+ 3832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.836026 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.836338 (+ 312us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:26.837914 (+ 1576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:26.837918 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:26.837921 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:26.837987 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:26.839724 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:26.840108 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:26.840112 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:26.840114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:26.840194 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:26.840470 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:26.840472 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:26.840474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:26.840652 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:26.840802 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:26.840998 (+ 196us) server_negotiation.cc:299] Negotiation successful
0620 19:10:26.841111 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1756,"thread_start_us":85,"threads_started":1}
I20260620 19:10:26.842866 17197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:26.820767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52032 (local address 127.9.155.126:44831)
0620 19:10:26.822077 (+ 1310us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:26.822081 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:26.822092 (+ 11us) server_negotiation.cc:407] Connection header received
0620 19:10:26.822126 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:26.822129 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:26.822172 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:26.822252 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:26.832361 (+ 10109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.834271 (+ 1910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.835630 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.835962 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.836060 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:26.839920 (+ 3860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:26.839923 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:26.839925 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:26.839946 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:26.841551 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:26.841940 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:26.841942 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:26.841944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:26.842062 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:26.842273 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:26.842275 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:26.842276 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:26.842432 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:26.842529 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:26.842666 (+ 137us) server_negotiation.cc:299] Negotiation successful
0620 19:10:26.842756 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1235,"thread_start_us":83,"threads_started":1}
I20260620 19:10:26.845602 15942 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52032:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:10:26.849000 15942 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:10:26.851877 17056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d022764bb4a442e8466cb260239c7e5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6d022764bb4a442e8466cb260239c7e5" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6d022764bb4a442e8466cb260239c7e5" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 44831 } } peers { permanent_uuid: "c20a27597512484cb7546601dada9ca9" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 35509 } attrs { promote: false } } peers { permanent_uuid: "12c2fb5fc9eb4fe2850ab1444aee8eda" member_type: NON_VOTER last_known_addr { host: "127.9.155.124" port: 46355 } attrs { promote: true } } }
I20260620 19:10:26.852039 17056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d022764bb4a442e8466cb260239c7e5 [sys.catalog]: This master's current role is: LEADER
I20260620 19:10:26.855278 17203 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 6d022764bb4a442e8466cb260239c7e5: attempting to promote NON_VOTER 12c2fb5fc9eb4fe2850ab1444aee8eda to VOTER
I20260620 19:10:26.855676 17203 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d022764bb4a442e8466cb260239c7e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6d022764bb4a442e8466cb260239c7e5" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 44831 } } peers { permanent_uuid: "c20a27597512484cb7546601dada9ca9" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 35509 } attrs { promote: false } } peers { permanent_uuid: "12c2fb5fc9eb4fe2850ab1444aee8eda" member_type: VOTER last_known_addr { host: "127.9.155.124" port: 46355 } attrs { promote: false } }
I20260620 19:10:26.856832 16554 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c20a27597512484cb7546601dada9ca9 [term 1 FOLLOWER]: Refusing update from remote peer 6d022764bb4a442e8466cb260239c7e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260620 19:10:26.857383 17201 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6d022764bb4a442e8466cb260239c7e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c20a27597512484cb7546601dada9ca9" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 35509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260620 19:10:26.857774 16738 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 12c2fb5fc9eb4fe2850ab1444aee8eda [term 1 LEARNER]: Refusing update from remote peer 6d022764bb4a442e8466cb260239c7e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260620 19:10:26.858184 17056 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6d022764bb4a442e8466cb260239c7e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12c2fb5fc9eb4fe2850ab1444aee8eda" member_type: VOTER last_known_addr { host: "127.9.155.124" port: 46355 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260620 19:10:26.860342 17056 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6d022764bb4a442e8466cb260239c7e5 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 12c2fb5fc9eb4fe2850ab1444aee8eda (127.9.155.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6d022764bb4a442e8466cb260239c7e5" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 44831 } } peers { permanent_uuid: "c20a27597512484cb7546601dada9ca9" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 35509 } attrs { promote: false } } peers { permanent_uuid: "12c2fb5fc9eb4fe2850ab1444aee8eda" member_type: VOTER last_known_addr { host: "127.9.155.124" port: 46355 } attrs { promote: false } } }
I20260620 19:10:26.860670 16554 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c20a27597512484cb7546601dada9ca9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 12c2fb5fc9eb4fe2850ab1444aee8eda (127.9.155.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6d022764bb4a442e8466cb260239c7e5" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 44831 } } peers { permanent_uuid: "c20a27597512484cb7546601dada9ca9" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 35509 } attrs { promote: false } } peers { permanent_uuid: "12c2fb5fc9eb4fe2850ab1444aee8eda" member_type: VOTER last_known_addr { host: "127.9.155.124" port: 46355 } attrs { promote: false } } }
I20260620 19:10:26.861835 17202 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d022764bb4a442e8466cb260239c7e5 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6d022764bb4a442e8466cb260239c7e5" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6d022764bb4a442e8466cb260239c7e5" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 44831 } } peers { permanent_uuid: "c20a27597512484cb7546601dada9ca9" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 35509 } attrs { promote: false } } peers { permanent_uuid: "12c2fb5fc9eb4fe2850ab1444aee8eda" member_type: VOTER last_known_addr { host: "127.9.155.124" port: 46355 } attrs { promote: false } } }
I20260620 19:10:26.861881 17056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d022764bb4a442e8466cb260239c7e5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6d022764bb4a442e8466cb260239c7e5" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6d022764bb4a442e8466cb260239c7e5" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 44831 } } peers { permanent_uuid: "c20a27597512484cb7546601dada9ca9" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 35509 } attrs { promote: false } } peers { permanent_uuid: "12c2fb5fc9eb4fe2850ab1444aee8eda" member_type: VOTER last_known_addr { host: "127.9.155.124" port: 46355 } attrs { promote: false } } }
I20260620 19:10:26.862030 17056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d022764bb4a442e8466cb260239c7e5 [sys.catalog]: This master's current role is: LEADER
I20260620 19:10:26.862154 17056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d022764bb4a442e8466cb260239c7e5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6d022764bb4a442e8466cb260239c7e5" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6d022764bb4a442e8466cb260239c7e5" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 44831 } } peers { permanent_uuid: "c20a27597512484cb7546601dada9ca9" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 35509 } attrs { promote: false } } peers { permanent_uuid: "12c2fb5fc9eb4fe2850ab1444aee8eda" member_type: VOTER last_known_addr { host: "127.9.155.124" port: 46355 } attrs { promote: false } } }
I20260620 19:10:26.862226 17056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d022764bb4a442e8466cb260239c7e5 [sys.catalog]: This master's current role is: LEADER
I20260620 19:10:26.862425 17202 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d022764bb4a442e8466cb260239c7e5 [sys.catalog]: This master's current role is: LEADER
I20260620 19:10:26.862267 16738 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 12c2fb5fc9eb4fe2850ab1444aee8eda [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 12c2fb5fc9eb4fe2850ab1444aee8eda (127.9.155.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6d022764bb4a442e8466cb260239c7e5" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 44831 } } peers { permanent_uuid: "c20a27597512484cb7546601dada9ca9" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 35509 } attrs { promote: false } } peers { permanent_uuid: "12c2fb5fc9eb4fe2850ab1444aee8eda" member_type: VOTER last_known_addr { host: "127.9.155.124" port: 46355 } attrs { promote: false } } }
I20260620 19:10:26.862639 17211 sys_catalog.cc:455] T 00000000000000000000000000000000 P c20a27597512484cb7546601dada9ca9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6d022764bb4a442e8466cb260239c7e5" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6d022764bb4a442e8466cb260239c7e5" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 44831 } } peers { permanent_uuid: "c20a27597512484cb7546601dada9ca9" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 35509 } attrs { promote: false } } peers { permanent_uuid: "12c2fb5fc9eb4fe2850ab1444aee8eda" member_type: VOTER last_known_addr { host: "127.9.155.124" port: 46355 } attrs { promote: false } } }
I20260620 19:10:26.862757 17211 sys_catalog.cc:458] T 00000000000000000000000000000000 P c20a27597512484cb7546601dada9ca9 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:10:26.864120 16768 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12c2fb5fc9eb4fe2850ab1444aee8eda [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6d022764bb4a442e8466cb260239c7e5" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6d022764bb4a442e8466cb260239c7e5" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 44831 } } peers { permanent_uuid: "c20a27597512484cb7546601dada9ca9" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 35509 } attrs { promote: false } } peers { permanent_uuid: "12c2fb5fc9eb4fe2850ab1444aee8eda" member_type: VOTER last_known_addr { host: "127.9.155.124" port: 46355 } attrs { promote: false } } }
I20260620 19:10:26.864267 16768 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12c2fb5fc9eb4fe2850ab1444aee8eda [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:10:26.870497 17214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:26.865452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.65:42443 (local address 127.0.0.1:40618)
0620 19:10:26.866546 (+ 1094us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:26.866559 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:26.866657 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:26.867103 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:26.867105 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:26.867120 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:10:26.867220 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:26.867226 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.868263 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.868266 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:26.869212 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:26.869221 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.870095 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.870099 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:26.870185 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:26.870208 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 19:10:26.870267 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 19:10:26.870344 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1034,"thread_start_us":87,"threads_started":1}
I20260620 19:10:26.870515 17216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:26.866181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40618 (local address 127.9.155.65:42443)
0620 19:10:26.866487 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:26.866498 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:26.866642 (+ 144us) server_negotiation.cc:407] Connection header received
0620 19:10:26.866767 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:26.866773 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:26.866985 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:26.867124 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:10:26.867304 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.868124 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.869332 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.869983 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.870082 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:26.870240 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:26.870294 (+ 54us) server_negotiation.cc:299] Negotiation successful
0620 19:10:26.870396 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":108,"threads_started":1}
I20260620 19:10:26.871142 17057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:26.864543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.67:34277 (local address 127.0.0.1:39544)
0620 19:10:26.864649 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:26.864665 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:26.864745 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:26.866703 (+ 1958us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:26.866706 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:26.866724 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:10:26.866831 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:26.866839 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.868761 (+ 1922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.868764 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:26.869698 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:26.869707 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.870793 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.870797 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:26.870885 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:26.870907 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 19:10:26.870982 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 19:10:26.871051 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25,"mutex_wait_us":30}
I20260620 19:10:26.871248 17212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:26.865043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39544 (local address 127.9.155.67:34277)
0620 19:10:26.865383 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:26.865396 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:26.865421 (+ 25us) server_negotiation.cc:407] Connection header received
0620 19:10:26.866341 (+ 920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:26.866349 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:26.866571 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:26.866721 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:10:26.867423 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.868634 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.869820 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.870635 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.870773 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:26.870910 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:26.871009 (+ 99us) server_negotiation.cc:299] Negotiation successful
0620 19:10:26.871102 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":107,"threads_started":1}
I20260620 19:10:26.874351 17117 tablet_service.cc:1467] Processing CreateTablet for tablet 6605c357e7a245ce8ec67cc91851aac3 (DEFAULT_TABLE table=test-table [id=6f4b538f055a417f8c38297cc0ec1e9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:10:26.874351 16842 tablet_service.cc:1467] Processing CreateTablet for tablet 6605c357e7a245ce8ec67cc91851aac3 (DEFAULT_TABLE table=test-table [id=6f4b538f055a417f8c38297cc0ec1e9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:10:26.875500 16842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6605c357e7a245ce8ec67cc91851aac3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:26.875500 17117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6605c357e7a245ce8ec67cc91851aac3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:26.880610 17213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:26.865806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.66:44711 (local address 127.0.0.1:45918)
0620 19:10:26.866833 (+ 1027us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:26.866845 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:26.866925 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:26.867274 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:26.867277 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:26.867293 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:10:26.867385 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:26.867391 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.868761 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.868764 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:26.870812 (+ 2048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:26.870821 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.871978 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.871982 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:26.872069 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:26.872091 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 19:10:26.880340 (+ 8249us) client_negotiation.cc:240] Negotiation successful
0620 19:10:26.880453 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":972,"mutex_wait_us":67,"thread_start_us":89,"threads_started":1}
I20260620 19:10:26.881300 17215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:26.865963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45918 (local address 127.9.155.66:44711)
0620 19:10:26.866405 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:26.866416 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:26.866932 (+ 516us) server_negotiation.cc:407] Connection header received
0620 19:10:26.867014 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:26.867017 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:26.867154 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:26.867282 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:10:26.867511 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.868653 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.870940 (+ 2287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.871700 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.871797 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:26.871916 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:26.881005 (+ 9089us) server_negotiation.cc:299] Negotiation successful
0620 19:10:26.881144 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":114,"threads_started":1}
I20260620 19:10:26.883947 17217 tablet_bootstrap.cc:492] T 6605c357e7a245ce8ec67cc91851aac3 P a2b48205cf584ab18dd060cf238791ab: Bootstrap starting.
I20260620 19:10:26.884091 16979 tablet_service.cc:1467] Processing CreateTablet for tablet 6605c357e7a245ce8ec67cc91851aac3 (DEFAULT_TABLE table=test-table [id=6f4b538f055a417f8c38297cc0ec1e9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:10:26.885227 16979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6605c357e7a245ce8ec67cc91851aac3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:26.886315 17217 tablet_bootstrap.cc:654] T 6605c357e7a245ce8ec67cc91851aac3 P a2b48205cf584ab18dd060cf238791ab: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:26.887192 17217 log.cc:824] T 6605c357e7a245ce8ec67cc91851aac3 P a2b48205cf584ab18dd060cf238791ab: Log is configured to *not* fsync() on all Append() calls
I20260620 19:10:26.887923 17218 tablet_bootstrap.cc:492] T 6605c357e7a245ce8ec67cc91851aac3 P af15d4c4a9304cc8a20af15051256305: Bootstrap starting.
I20260620 19:10:26.889422 17217 tablet_bootstrap.cc:492] T 6605c357e7a245ce8ec67cc91851aac3 P a2b48205cf584ab18dd060cf238791ab: No bootstrap required, opened a new log
I20260620 19:10:26.889667 17217 ts_tablet_manager.cc:1397] T 6605c357e7a245ce8ec67cc91851aac3 P a2b48205cf584ab18dd060cf238791ab: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 19:10:26.890205 17218 tablet_bootstrap.cc:654] T 6605c357e7a245ce8ec67cc91851aac3 P af15d4c4a9304cc8a20af15051256305: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:26.891040 17218 log.cc:824] T 6605c357e7a245ce8ec67cc91851aac3 P af15d4c4a9304cc8a20af15051256305: Log is configured to *not* fsync() on all Append() calls
I20260620 19:10:26.893129 17218 tablet_bootstrap.cc:492] T 6605c357e7a245ce8ec67cc91851aac3 P af15d4c4a9304cc8a20af15051256305: No bootstrap required, opened a new log
I20260620 19:10:26.893316 17218 ts_tablet_manager.cc:1397] T 6605c357e7a245ce8ec67cc91851aac3 P af15d4c4a9304cc8a20af15051256305: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 19:10:26.893745 17217 raft_consensus.cc:348] T 6605c357e7a245ce8ec67cc91851aac3 P a2b48205cf584ab18dd060cf238791ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b48205cf584ab18dd060cf238791ab" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34277 } } peers { permanent_uuid: "af15d4c4a9304cc8a20af15051256305" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 42443 } } peers { permanent_uuid: "22fadda898e54ffd9a8eaac7a55fcb91" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 44711 } }
I20260620 19:10:26.894080 17217 raft_consensus.cc:374] T 6605c357e7a245ce8ec67cc91851aac3 P a2b48205cf584ab18dd060cf238791ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:26.894133 17217 raft_consensus.cc:729] T 6605c357e7a245ce8ec67cc91851aac3 P a2b48205cf584ab18dd060cf238791ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2b48205cf584ab18dd060cf238791ab, State: Initialized, Role: FOLLOWER
I20260620 19:10:26.894328 17221 tablet_bootstrap.cc:492] T 6605c357e7a245ce8ec67cc91851aac3 P 22fadda898e54ffd9a8eaac7a55fcb91: Bootstrap starting.
I20260620 19:10:26.894588 17217 consensus_queue.cc:260] T 6605c357e7a245ce8ec67cc91851aac3 P a2b48205cf584ab18dd060cf238791ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b48205cf584ab18dd060cf238791ab" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34277 } } peers { permanent_uuid: "af15d4c4a9304cc8a20af15051256305" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 42443 } } peers { permanent_uuid: "22fadda898e54ffd9a8eaac7a55fcb91" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 44711 } }
I20260620 19:10:26.895555 17217 ts_tablet_manager.cc:1428] T 6605c357e7a245ce8ec67cc91851aac3 P a2b48205cf584ab18dd060cf238791ab: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 19:10:26.895920 17184 heartbeater.cc:503] Master 127.9.155.126:44831 was elected leader, sending a full tablet report...
I20260620 19:10:26.897691 17218 raft_consensus.cc:348] T 6605c357e7a245ce8ec67cc91851aac3 P af15d4c4a9304cc8a20af15051256305 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b48205cf584ab18dd060cf238791ab" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34277 } } peers { permanent_uuid: "af15d4c4a9304cc8a20af15051256305" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 42443 } } peers { permanent_uuid: "22fadda898e54ffd9a8eaac7a55fcb91" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 44711 } }
I20260620 19:10:26.897959 17218 raft_consensus.cc:374] T 6605c357e7a245ce8ec67cc91851aac3 P af15d4c4a9304cc8a20af15051256305 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:26.898006 17218 raft_consensus.cc:729] T 6605c357e7a245ce8ec67cc91851aac3 P af15d4c4a9304cc8a20af15051256305 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af15d4c4a9304cc8a20af15051256305, State: Initialized, Role: FOLLOWER
I20260620 19:10:26.898422 17218 consensus_queue.cc:260] T 6605c357e7a245ce8ec67cc91851aac3 P af15d4c4a9304cc8a20af15051256305 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b48205cf584ab18dd060cf238791ab" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34277 } } peers { permanent_uuid: "af15d4c4a9304cc8a20af15051256305" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 42443 } } peers { permanent_uuid: "22fadda898e54ffd9a8eaac7a55fcb91" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 44711 } }
I20260620 19:10:26.898550 17221 tablet_bootstrap.cc:654] T 6605c357e7a245ce8ec67cc91851aac3 P 22fadda898e54ffd9a8eaac7a55fcb91: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:26.899391 17218 ts_tablet_manager.cc:1428] T 6605c357e7a245ce8ec67cc91851aac3 P af15d4c4a9304cc8a20af15051256305: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 19:10:26.899384 17221 log.cc:824] T 6605c357e7a245ce8ec67cc91851aac3 P 22fadda898e54ffd9a8eaac7a55fcb91: Log is configured to *not* fsync() on all Append() calls
I20260620 19:10:26.899760 16909 heartbeater.cc:503] Master 127.9.155.126:44831 was elected leader, sending a full tablet report...
I20260620 19:10:26.903455 17221 tablet_bootstrap.cc:492] T 6605c357e7a245ce8ec67cc91851aac3 P 22fadda898e54ffd9a8eaac7a55fcb91: No bootstrap required, opened a new log
I20260620 19:10:26.903653 17221 ts_tablet_manager.cc:1397] T 6605c357e7a245ce8ec67cc91851aac3 P 22fadda898e54ffd9a8eaac7a55fcb91: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260620 19:10:26.907084 17221 raft_consensus.cc:348] T 6605c357e7a245ce8ec67cc91851aac3 P 22fadda898e54ffd9a8eaac7a55fcb91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b48205cf584ab18dd060cf238791ab" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34277 } } peers { permanent_uuid: "af15d4c4a9304cc8a20af15051256305" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 42443 } } peers { permanent_uuid: "22fadda898e54ffd9a8eaac7a55fcb91" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 44711 } }
I20260620 19:10:26.907328 17221 raft_consensus.cc:374] T 6605c357e7a245ce8ec67cc91851aac3 P 22fadda898e54ffd9a8eaac7a55fcb91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:26.907377 17221 raft_consensus.cc:729] T 6605c357e7a245ce8ec67cc91851aac3 P 22fadda898e54ffd9a8eaac7a55fcb91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22fadda898e54ffd9a8eaac7a55fcb91, State: Initialized, Role: FOLLOWER
I20260620 19:10:26.907752 17221 consensus_queue.cc:260] T 6605c357e7a245ce8ec67cc91851aac3 P 22fadda898e54ffd9a8eaac7a55fcb91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b48205cf584ab18dd060cf238791ab" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34277 } } peers { permanent_uuid: "af15d4c4a9304cc8a20af15051256305" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 42443 } } peers { permanent_uuid: "22fadda898e54ffd9a8eaac7a55fcb91" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 44711 } }
I20260620 19:10:26.908532 17046 heartbeater.cc:503] Master 127.9.155.126:44831 was elected leader, sending a full tablet report...
I20260620 19:10:26.908768 17221 ts_tablet_manager.cc:1428] T 6605c357e7a245ce8ec67cc91851aac3 P 22fadda898e54ffd9a8eaac7a55fcb91: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260620 19:10:26.927773 16911 tablet.cc:2369] T 6605c357e7a245ce8ec67cc91851aac3 P af15d4c4a9304cc8a20af15051256305: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:10:26.989033 17223 raft_consensus.cc:482] T 6605c357e7a245ce8ec67cc91851aac3 P af15d4c4a9304cc8a20af15051256305 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:10:26.989260 17223 raft_consensus.cc:504] T 6605c357e7a245ce8ec67cc91851aac3 P af15d4c4a9304cc8a20af15051256305 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b48205cf584ab18dd060cf238791ab" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34277 } } peers { permanent_uuid: "af15d4c4a9304cc8a20af15051256305" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 42443 } } peers { permanent_uuid: "22fadda898e54ffd9a8eaac7a55fcb91" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 44711 } }
I20260620 19:10:26.990463 17223 leader_election.cc:288] T 6605c357e7a245ce8ec67cc91851aac3 P af15d4c4a9304cc8a20af15051256305 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2b48205cf584ab18dd060cf238791ab (127.9.155.67:34277), 22fadda898e54ffd9a8eaac7a55fcb91 (127.9.155.66:44711)
I20260620 19:10:26.994387 17212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:26.990835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.65:46665 (local address 127.9.155.67:34277)
0620 19:10:26.991006 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:26.991010 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:26.991095 (+ 85us) server_negotiation.cc:407] Connection header received
0620 19:10:26.991264 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:26.991268 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:26.991323 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:26.991405 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:10:26.991676 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.992179 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.993284 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.993775 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.993870 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:26.993933 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:26.994168 (+ 235us) server_negotiation.cc:299] Negotiation successful
0620 19:10:26.994243 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 19:10:26.994374 17226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:26.990692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.67:34277 (local address 127.9.155.65:46665)
0620 19:10:26.991017 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:26.991033 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:26.991151 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:26.991434 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:26.991437 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:26.991456 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:10:26.991571 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:26.991577 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.992297 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.992300 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:26.993150 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:26.993162 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.993905 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.993908 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:26.994029 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:26.994058 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 19:10:26.994126 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 19:10:26.994205 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":21,"thread_start_us":124,"threads_started":1}
I20260620 19:10:26.994565 17227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:26.991127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.66:44711 (local address 127.9.155.65:54085)
0620 19:10:26.991342 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:26.991354 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:26.991474 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:26.991699 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:26.991702 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:26.991730 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:10:26.991834 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:26.991839 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.992597 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.992600 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:26.993421 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:26.993432 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.994106 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.994110 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:26.994192 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:26.994214 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 19:10:26.994285 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 19:10:26.994352 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":1,"thread_start_us":80,"threads_started":1}
I20260620 19:10:26.994561 17215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:26.991249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.65:54085 (local address 127.9.155.66:44711)
0620 19:10:26.991368 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:26.991373 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:26.991388 (+ 15us) server_negotiation.cc:407] Connection header received
0620 19:10:26.991576 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:26.991579 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:26.991621 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:26.991683 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:10:26.991952 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.992498 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.993543 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:26.994003 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:26.994118 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:26.994164 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:26.994357 (+ 193us) server_negotiation.cc:299] Negotiation successful
0620 19:10:26.994434 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 19:10:26.995276 16999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6605c357e7a245ce8ec67cc91851aac3" candidate_uuid: "af15d4c4a9304cc8a20af15051256305" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fadda898e54ffd9a8eaac7a55fcb91" is_pre_election: true
I20260620 19:10:26.995611 16999 raft_consensus.cc:2454] T 6605c357e7a245ce8ec67cc91851aac3 P 22fadda898e54ffd9a8eaac7a55fcb91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af15d4c4a9304cc8a20af15051256305 in term 0.
I20260620 19:10:26.996081 17137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6605c357e7a245ce8ec67cc91851aac3" candidate_uuid: "af15d4c4a9304cc8a20af15051256305" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2b48205cf584ab18dd060cf238791ab" is_pre_election: true
I20260620 19:10:26.996102 16796 leader_election.cc:302] T 6605c357e7a245ce8ec67cc91851aac3 P af15d4c4a9304cc8a20af15051256305 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fadda898e54ffd9a8eaac7a55fcb91, af15d4c4a9304cc8a20af15051256305; no voters:
I20260620 19:10:26.996377 17137 raft_consensus.cc:2454] T 6605c357e7a245ce8ec67cc91851aac3 P a2b48205cf584ab18dd060cf238791ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af15d4c4a9304cc8a20af15051256305 in term 0.
I20260620 19:10:26.996430 17223 raft_consensus.cc:2781] T 6605c357e7a245ce8ec67cc91851aac3 P af15d4c4a9304cc8a20af15051256305 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:10:26.996537 17223 raft_consensus.cc:482] T 6605c357e7a245ce8ec67cc91851aac3 P af15d4c4a9304cc8a20af15051256305 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:10:26.996588 17223 raft_consensus.cc:3037] T 6605c357e7a245ce8ec67cc91851aac3 P af15d4c4a9304cc8a20af15051256305 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:26.997673 17223 raft_consensus.cc:504] T 6605c357e7a245ce8ec67cc91851aac3 P af15d4c4a9304cc8a20af15051256305 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b48205cf584ab18dd060cf238791ab" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34277 } } peers { permanent_uuid: "af15d4c4a9304cc8a20af15051256305" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 42443 } } peers { permanent_uuid: "22fadda898e54ffd9a8eaac7a55fcb91" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 44711 } }
I20260620 19:10:26.998035 17223 leader_election.cc:288] T 6605c357e7a245ce8ec67cc91851aac3 P af15d4c4a9304cc8a20af15051256305 [CANDIDATE]: Term 1 election: Requested vote from peers a2b48205cf584ab18dd060cf238791ab (127.9.155.67:34277), 22fadda898e54ffd9a8eaac7a55fcb91 (127.9.155.66:44711)
I20260620 19:10:26.998426 17137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6605c357e7a245ce8ec67cc91851aac3" candidate_uuid: "af15d4c4a9304cc8a20af15051256305" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2b48205cf584ab18dd060cf238791ab"
I20260620 19:10:26.998440 16999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6605c357e7a245ce8ec67cc91851aac3" candidate_uuid: "af15d4c4a9304cc8a20af15051256305" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fadda898e54ffd9a8eaac7a55fcb91"
I20260620 19:10:26.998570 16999 raft_consensus.cc:3037] T 6605c357e7a245ce8ec67cc91851aac3 P 22fadda898e54ffd9a8eaac7a55fcb91 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:26.998569 17137 raft_consensus.cc:3037] T 6605c357e7a245ce8ec67cc91851aac3 P a2b48205cf584ab18dd060cf238791ab [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:26.999884 17137 raft_consensus.cc:2454] T 6605c357e7a245ce8ec67cc91851aac3 P a2b48205cf584ab18dd060cf238791ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af15d4c4a9304cc8a20af15051256305 in term 1.
I20260620 19:10:26.999897 16999 raft_consensus.cc:2454] T 6605c357e7a245ce8ec67cc91851aac3 P 22fadda898e54ffd9a8eaac7a55fcb91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af15d4c4a9304cc8a20af15051256305 in term 1.
I20260620 19:10:27.000183 16796 leader_election.cc:302] T 6605c357e7a245ce8ec67cc91851aac3 P af15d4c4a9304cc8a20af15051256305 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2b48205cf584ab18dd060cf238791ab, af15d4c4a9304cc8a20af15051256305; no voters:
I20260620 19:10:27.000379 17223 raft_consensus.cc:2781] T 6605c357e7a245ce8ec67cc91851aac3 P af15d4c4a9304cc8a20af15051256305 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:10:27.000579 17223 raft_consensus.cc:686] T 6605c357e7a245ce8ec67cc91851aac3 P af15d4c4a9304cc8a20af15051256305 [term 1 LEADER]: Becoming Leader. State: Replica: af15d4c4a9304cc8a20af15051256305, State: Running, Role: LEADER
I20260620 19:10:27.000931 17223 consensus_queue.cc:237] T 6605c357e7a245ce8ec67cc91851aac3 P af15d4c4a9304cc8a20af15051256305 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b48205cf584ab18dd060cf238791ab" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34277 } } peers { permanent_uuid: "af15d4c4a9304cc8a20af15051256305" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 42443 } } peers { permanent_uuid: "22fadda898e54ffd9a8eaac7a55fcb91" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 44711 } }
I20260620 19:10:27.004201 15943 catalog_manager.cc:5495] T 6605c357e7a245ce8ec67cc91851aac3 P af15d4c4a9304cc8a20af15051256305 reported cstate change: term changed from 0 to 1, leader changed from <none> to af15d4c4a9304cc8a20af15051256305 (127.9.155.65). New cstate: current_term: 1 leader_uuid: "af15d4c4a9304cc8a20af15051256305" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b48205cf584ab18dd060cf238791ab" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34277 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af15d4c4a9304cc8a20af15051256305" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 42443 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22fadda898e54ffd9a8eaac7a55fcb91" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 44711 } health_report { overall_health: UNKNOWN } } }
I20260620 19:10:27.025024 17216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:27.021543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40620 (local address 127.9.155.65:42443)
0620 19:10:27.021664 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:27.021669 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:27.021684 (+ 15us) server_negotiation.cc:407] Connection header received
0620 19:10:27.021773 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:27.021775 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:27.021827 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:27.021927 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:10:27.022161 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:27.022675 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:27.023537 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:27.023750 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:27.023836 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:27.024121 (+ 285us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:10:27.024629 (+ 508us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:10:27.024738 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:27.024813 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 19:10:27.024885 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 19:10:27.030819 16999 raft_consensus.cc:1261] T 6605c357e7a245ce8ec67cc91851aac3 P 22fadda898e54ffd9a8eaac7a55fcb91 [term 1 FOLLOWER]: Refusing update from remote peer af15d4c4a9304cc8a20af15051256305: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:10:27.030827 17137 raft_consensus.cc:1261] T 6605c357e7a245ce8ec67cc91851aac3 P a2b48205cf584ab18dd060cf238791ab [term 1 FOLLOWER]: Refusing update from remote peer af15d4c4a9304cc8a20af15051256305: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:10:27.031548 17223 consensus_queue.cc:1035] T 6605c357e7a245ce8ec67cc91851aac3 P af15d4c4a9304cc8a20af15051256305 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b48205cf584ab18dd060cf238791ab" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:10:27.031764 17228 consensus_queue.cc:1035] T 6605c357e7a245ce8ec67cc91851aac3 P af15d4c4a9304cc8a20af15051256305 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fadda898e54ffd9a8eaac7a55fcb91" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 44711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:10:27.041850 17233 mvcc.cc:205] Tried to move back new op lower bound from 7299000840311119872 to 7299000840198463488. Current Snapshot: MvccSnapshot[applied={T|T < 7299000840311119872}]
I20260620 19:10:27.044183 17232 mvcc.cc:205] Tried to move back new op lower bound from 7299000840311119872 to 7299000840198463488. Current Snapshot: MvccSnapshot[applied={T|T < 7299000840311119872 or (T in {7299000840311119872})}]
I20260620 19:10:27.052452 15942 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:10:27.052611 15942 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:10:27.057957 15942 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6d022764bb4a442e8466cb260239c7e5: Sending DeleteTablet for 3 replicas of tablet 6605c357e7a245ce8ec67cc91851aac3
I20260620 19:10:27.058858 17117 tablet_service.cc:1514] Processing DeleteTablet for tablet 6605c357e7a245ce8ec67cc91851aac3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:10:27 UTC) from {username='oryx', principal='oryx/127.9.155.126@KRBTEST.COM'} at 127.0.0.1:39544
I20260620 19:10:27.058918 16979 tablet_service.cc:1514] Processing DeleteTablet for tablet 6605c357e7a245ce8ec67cc91851aac3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:10:27 UTC) from {username='oryx', principal='oryx/127.9.155.126@KRBTEST.COM'} at 127.0.0.1:45918
I20260620 19:10:27.059373 17242 tablet_replica.cc:330] T 6605c357e7a245ce8ec67cc91851aac3 P a2b48205cf584ab18dd060cf238791ab: stopping tablet replica
I20260620 19:10:27.059371 16842 tablet_service.cc:1514] Processing DeleteTablet for tablet 6605c357e7a245ce8ec67cc91851aac3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:10:27 UTC) from {username='oryx', principal='oryx/127.9.155.126@KRBTEST.COM'} at 127.0.0.1:40618
I20260620 19:10:27.059741 17242 raft_consensus.cc:2229] T 6605c357e7a245ce8ec67cc91851aac3 P a2b48205cf584ab18dd060cf238791ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:10:27.059881 17244 tablet_replica.cc:330] T 6605c357e7a245ce8ec67cc91851aac3 P af15d4c4a9304cc8a20af15051256305: stopping tablet replica
I20260620 19:10:27.060096 17242 raft_consensus.cc:2258] T 6605c357e7a245ce8ec67cc91851aac3 P a2b48205cf584ab18dd060cf238791ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:10:27.060292 17244 raft_consensus.cc:2229] T 6605c357e7a245ce8ec67cc91851aac3 P af15d4c4a9304cc8a20af15051256305 [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:10:27.060750 17244 raft_consensus.cc:2258] T 6605c357e7a245ce8ec67cc91851aac3 P af15d4c4a9304cc8a20af15051256305 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:10:27.062204 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 15911
I20260620 19:10:27.062844 17244 ts_tablet_manager.cc:1905] T 6605c357e7a245ce8ec67cc91851aac3 P af15d4c4a9304cc8a20af15051256305: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:10:27.063035 17242 ts_tablet_manager.cc:1905] T 6605c357e7a245ce8ec67cc91851aac3 P a2b48205cf584ab18dd060cf238791ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:10:27.065673 17242 ts_tablet_manager.cc:1918] T 6605c357e7a245ce8ec67cc91851aac3 P a2b48205cf584ab18dd060cf238791ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:10:27.065677 17244 ts_tablet_manager.cc:1918] T 6605c357e7a245ce8ec67cc91851aac3 P af15d4c4a9304cc8a20af15051256305: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:10:27.065775 17242 log.cc:1196] T 6605c357e7a245ce8ec67cc91851aac3 P a2b48205cf584ab18dd060cf238791ab: Deleting WAL directory at /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-2/wal/wals/6605c357e7a245ce8ec67cc91851aac3
I20260620 19:10:27.065915 17244 log.cc:1196] T 6605c357e7a245ce8ec67cc91851aac3 P af15d4c4a9304cc8a20af15051256305: Deleting WAL directory at /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-0/wal/wals/6605c357e7a245ce8ec67cc91851aac3
I20260620 19:10:27.066066 17242 ts_tablet_manager.cc:1939] T 6605c357e7a245ce8ec67cc91851aac3 P a2b48205cf584ab18dd060cf238791ab: Deleting consensus metadata
I20260620 19:10:27.066365 17244 ts_tablet_manager.cc:1939] T 6605c357e7a245ce8ec67cc91851aac3 P af15d4c4a9304cc8a20af15051256305: Deleting consensus metadata
I20260620 19:10:27.068926 17243 tablet_replica.cc:330] T 6605c357e7a245ce8ec67cc91851aac3 P 22fadda898e54ffd9a8eaac7a55fcb91: stopping tablet replica
W20260620 19:10:27.069165 16795 connection.cc:535] server connection from 127.0.0.1:40618 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 19:10:27.069299 17243 raft_consensus.cc:2229] T 6605c357e7a245ce8ec67cc91851aac3 P 22fadda898e54ffd9a8eaac7a55fcb91 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260620 19:10:27.069540 17073 connection.cc:535] server connection from 127.0.0.1:39544 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 19:10:27.069677 17243 raft_consensus.cc:2258] T 6605c357e7a245ce8ec67cc91851aac3 P 22fadda898e54ffd9a8eaac7a55fcb91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:10:27.071375 17243 ts_tablet_manager.cc:1905] T 6605c357e7a245ce8ec67cc91851aac3 P 22fadda898e54ffd9a8eaac7a55fcb91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:10:27.074348 17243 ts_tablet_manager.cc:1918] T 6605c357e7a245ce8ec67cc91851aac3 P 22fadda898e54ffd9a8eaac7a55fcb91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:10:27.074465 17243 log.cc:1196] T 6605c357e7a245ce8ec67cc91851aac3 P 22fadda898e54ffd9a8eaac7a55fcb91: Deleting WAL directory at /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-1/wal/wals/6605c357e7a245ce8ec67cc91851aac3
I20260620 19:10:27.074800 17243 ts_tablet_manager.cc:1939] T 6605c357e7a245ce8ec67cc91851aac3 P 22fadda898e54ffd9a8eaac7a55fcb91: Deleting consensus metadata
W20260620 19:10:27.076050 16934 connection.cc:441] server connection from 127.0.0.1:45918 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:45918 (request call id 1) could send its response
I20260620 19:10:27.079250 17198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:27.061484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34336 (local address 127.9.155.125:35509)
0620 19:10:27.061724 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:27.061729 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:27.068606 (+ 6877us) server_negotiation.cc:407] Connection header received
0620 19:10:27.068747 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:27.068750 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:27.068813 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:27.068898 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:27.070405 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:27.071238 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:27.072580 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:27.072939 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:27.073013 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:27.075200 (+ 2187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:27.075204 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:27.075207 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:27.075236 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:27.077107 (+ 1871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:27.077627 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:27.077631 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:27.077633 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:27.077718 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:27.078005 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:27.078009 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:27.078011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:27.078505 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:27.078607 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:27.078945 (+ 338us) server_negotiation.cc:299] Negotiation successful
0620 19:10:27.079082 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159}
I20260620 19:10:27.083177 16915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:27.073541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42990 (local address 127.9.155.124:46355)
0620 19:10:27.073650 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:27.073654 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:27.074506 (+ 852us) server_negotiation.cc:407] Connection header received
0620 19:10:27.074704 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:27.074707 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:27.074761 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:27.074850 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:27.076001 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:27.076620 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:27.077412 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:27.077706 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:27.077803 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:27.079481 (+ 1678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:27.079485 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:27.079488 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:27.079516 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:27.081163 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:27.081639 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:27.081643 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:27.081645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:27.081729 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:27.081997 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:27.082000 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:27.082002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:27.082204 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:27.082303 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:27.082648 (+ 345us) server_negotiation.cc:299] Negotiation successful
0620 19:10:27.082773 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 19:10:27.087384 16666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:27.077743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34338 (local address 127.9.155.125:35509)
0620 19:10:27.077868 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:27.077872 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:27.078183 (+ 311us) server_negotiation.cc:407] Connection header received
0620 19:10:27.078340 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:27.078343 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:27.078386 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:27.078447 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:27.079179 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:27.079943 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:27.082114 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:27.082427 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:27.082499 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:27.084286 (+ 1787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:27.084290 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:27.084293 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:27.084321 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:27.085604 (+ 1283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:27.086048 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:27.086052 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:27.086054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:27.086138 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:27.086483 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:27.086486 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:27.086488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:27.086711 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:27.086842 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:27.087092 (+ 250us) server_negotiation.cc:299] Negotiation successful
0620 19:10:27.087208 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 19:10:27.089243 17194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:27.061598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42978 (local address 127.9.155.124:46355)
0620 19:10:27.061822 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:27.061826 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:27.080328 (+ 18502us) server_negotiation.cc:407] Connection header received
0620 19:10:27.080499 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:27.080502 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:27.080549 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:27.080598 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:27.081394 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:27.082171 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:27.083575 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:27.083931 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:27.084003 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:27.085715 (+ 1712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:27.085719 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:27.085722 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:27.085746 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:27.087322 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:27.087816 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:27.087820 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:27.087822 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:27.087905 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:27.088180 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:27.088184 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:27.088186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:27.088447 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:27.088569 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:27.088989 (+ 420us) server_negotiation.cc:299] Negotiation successful
0620 19:10:27.089094 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147}
I20260620 19:10:28.043968 17252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:28.043486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:44831 (local address 127.9.155.66:58659)
0620 19:10:28.043759 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:28.043844 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:44831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":85,"threads_started":1}
W20260620 19:10:28.044348 17046 heartbeater.cc:644] Failed to heartbeat to 127.9.155.126:44831 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.9.155.126:44831: connect: Connection refused (error 111)
I20260620 19:10:28.045770 17253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:28.045230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:44831 (local address 127.9.155.65:42251)
0620 19:10:28.045561 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:28.045649 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:44831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":78,"threads_started":1}
W20260620 19:10:28.046090 16909 heartbeater.cc:644] Failed to heartbeat to 127.9.155.126:44831 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.9.155.126:44831: connect: Connection refused (error 111)
I20260620 19:10:28.047298 17254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:28.046507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:44831 (local address 127.9.155.67:50869)
0620 19:10:28.046769 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:28.047196 (+ 427us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:44831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":67,"threads_started":1}
W20260620 19:10:28.047549 17184 heartbeater.cc:644] Failed to heartbeat to 127.9.155.126:44831 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.9.155.126:44831: connect: Connection refused (error 111)
I20260620 19:10:28.557896 17255 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12c2fb5fc9eb4fe2850ab1444aee8eda [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6d022764bb4a442e8466cb260239c7e5)
I20260620 19:10:28.558048 17255 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12c2fb5fc9eb4fe2850ab1444aee8eda [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6d022764bb4a442e8466cb260239c7e5" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 44831 } } peers { permanent_uuid: "c20a27597512484cb7546601dada9ca9" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 35509 } attrs { promote: false } } peers { permanent_uuid: "12c2fb5fc9eb4fe2850ab1444aee8eda" member_type: VOTER last_known_addr { host: "127.9.155.124" port: 46355 } attrs { promote: false } }
I20260620 19:10:28.558355 17256 raft_consensus.cc:482] T 00000000000000000000000000000000 P c20a27597512484cb7546601dada9ca9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6d022764bb4a442e8466cb260239c7e5)
I20260620 19:10:28.558544 17256 raft_consensus.cc:504] T 00000000000000000000000000000000 P c20a27597512484cb7546601dada9ca9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6d022764bb4a442e8466cb260239c7e5" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 44831 } } peers { permanent_uuid: "c20a27597512484cb7546601dada9ca9" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 35509 } attrs { promote: false } } peers { permanent_uuid: "12c2fb5fc9eb4fe2850ab1444aee8eda" member_type: VOTER last_known_addr { host: "127.9.155.124" port: 46355 } attrs { promote: false } }
I20260620 19:10:28.558625 17255 leader_election.cc:288] T 00000000000000000000000000000000 P 12c2fb5fc9eb4fe2850ab1444aee8eda [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d022764bb4a442e8466cb260239c7e5 (127.9.155.126:44831), c20a27597512484cb7546601dada9ca9 (127.9.155.125:35509)
I20260620 19:10:28.559371 17257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:28.558814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:44831 (local address 127.0.0.1:38870)
0620 19:10:28.559171 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:28.559258 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:44831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":55,"threads_started":1}
I20260620 19:10:28.559609 17256 leader_election.cc:288] T 00000000000000000000000000000000 P c20a27597512484cb7546601dada9ca9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d022764bb4a442e8466cb260239c7e5 (127.9.155.126:44831), 12c2fb5fc9eb4fe2850ab1444aee8eda (127.9.155.124:46355)
W20260620 19:10:28.559707 16715 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.155.126:44831: connect: Connection refused (error 111)
I20260620 19:10:28.560189 17260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:28.559691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:44831 (local address 127.0.0.1:38872)
0620 19:10:28.560033 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:28.560105 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:44831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":52,"threads_started":1}
W20260620 19:10:28.560350 16530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.155.126:44831: connect: Connection refused (error 111)
I20260620 19:10:28.561050 17257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:28.560845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:44831 (local address 127.0.0.1:38882)
0620 19:10:28.560936 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:28.560985 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:44831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260620 19:10:28.561167 17260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:28.560934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:44831 (local address 127.0.0.1:38888)
0620 19:10:28.561067 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:28.561110 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:44831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260620 19:10:28.561190 16715 leader_election.cc:334] T 00000000000000000000000000000000 P 12c2fb5fc9eb4fe2850ab1444aee8eda [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6d022764bb4a442e8466cb260239c7e5 (127.9.155.126:44831): Network error: Client connection negotiation failed: client connection to 127.9.155.126:44831: connect: Connection refused (error 111)
W20260620 19:10:28.561276 16530 leader_election.cc:334] T 00000000000000000000000000000000 P c20a27597512484cb7546601dada9ca9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6d022764bb4a442e8466cb260239c7e5 (127.9.155.126:44831): Network error: Client connection negotiation failed: client connection to 127.9.155.126:44831: connect: Connection refused (error 111)
I20260620 19:10:28.563346 17258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:28.558853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.125:35509 (local address 127.0.0.1:54200)
0620 19:10:28.559247 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:28.559258 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:28.559360 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:28.559701 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:28.559704 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:28.559725 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:10:28.559823 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:28.559830 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:28.560827 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:28.560831 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:28.562169 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:28.562176 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:28.563045 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:28.563047 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:28.563114 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:28.563134 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 19:10:28.563192 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 19:10:28.563246 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":75,"threads_started":1}
I20260620 19:10:28.563409 17259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:28.558934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54200 (local address 127.9.155.125:35509)
0620 19:10:28.559167 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:28.559172 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:28.559322 (+ 150us) server_negotiation.cc:407] Connection header received
0620 19:10:28.559466 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:28.559470 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:28.559511 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:28.559577 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:10:28.559914 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:28.560689 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:28.562263 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:28.562930 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:28.563044 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:28.563103 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:28.563217 (+ 114us) server_negotiation.cc:299] Negotiation successful
0620 19:10:28.563285 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":156,"thread_start_us":59,"threads_started":1}
I20260620 19:10:28.563766 16554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "12c2fb5fc9eb4fe2850ab1444aee8eda" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c20a27597512484cb7546601dada9ca9" is_pre_election: true
I20260620 19:10:28.563901 16554 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c20a27597512484cb7546601dada9ca9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12c2fb5fc9eb4fe2850ab1444aee8eda in term 1.
I20260620 19:10:28.564179 16713 leader_election.cc:302] T 00000000000000000000000000000000 P 12c2fb5fc9eb4fe2850ab1444aee8eda [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 12c2fb5fc9eb4fe2850ab1444aee8eda, c20a27597512484cb7546601dada9ca9; no voters: 6d022764bb4a442e8466cb260239c7e5
I20260620 19:10:28.564427 17255 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12c2fb5fc9eb4fe2850ab1444aee8eda [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 19:10:28.564474 17255 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12c2fb5fc9eb4fe2850ab1444aee8eda [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6d022764bb4a442e8466cb260239c7e5)
I20260620 19:10:28.564518 17255 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12c2fb5fc9eb4fe2850ab1444aee8eda [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:10:28.564641 17262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:28.559896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.124:46355 (local address 127.0.0.1:33948)
0620 19:10:28.560117 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:28.560130 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:28.560235 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:28.561102 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:28.561105 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:28.561122 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:10:28.561213 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:28.561219 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:28.562223 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:28.562227 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:28.563487 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:28.563497 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:28.564269 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:28.564272 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:28.564360 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:28.564381 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 19:10:28.564444 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 19:10:28.564524 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":59,"thread_start_us":61,"threads_started":1}
I20260620 19:10:28.564639 17264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:28.560448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33948 (local address 127.9.155.124:46355)
0620 19:10:28.560668 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:28.560677 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:28.560688 (+ 11us) server_negotiation.cc:407] Connection header received
0620 19:10:28.560938 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:28.560941 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:28.560982 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:28.561310 (+ 328us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:10:28.561397 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:28.562118 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:28.563582 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:28.564148 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:28.564263 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:28.564315 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:28.564466 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 19:10:28.564531 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":96,"threads_started":1}
I20260620 19:10:28.565129 16738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c20a27597512484cb7546601dada9ca9" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "12c2fb5fc9eb4fe2850ab1444aee8eda" is_pre_election: true
I20260620 19:10:28.565639 17255 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12c2fb5fc9eb4fe2850ab1444aee8eda [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6d022764bb4a442e8466cb260239c7e5" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 44831 } } peers { permanent_uuid: "c20a27597512484cb7546601dada9ca9" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 35509 } attrs { promote: false } } peers { permanent_uuid: "12c2fb5fc9eb4fe2850ab1444aee8eda" member_type: VOTER last_known_addr { host: "127.9.155.124" port: 46355 } attrs { promote: false } }
I20260620 19:10:28.565874 16738 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 12c2fb5fc9eb4fe2850ab1444aee8eda [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c20a27597512484cb7546601dada9ca9 in current term 2: Already voted for candidate 12c2fb5fc9eb4fe2850ab1444aee8eda in this term.
I20260620 19:10:28.566260 17258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:28.566066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:44831 (local address 127.0.0.1:38902)
0620 19:10:28.566172 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:28.566214 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:44831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260620 19:10:28.566349 16531 leader_election.cc:302] T 00000000000000000000000000000000 P c20a27597512484cb7546601dada9ca9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c20a27597512484cb7546601dada9ca9; no voters: 12c2fb5fc9eb4fe2850ab1444aee8eda, 6d022764bb4a442e8466cb260239c7e5
I20260620 19:10:28.566286 17255 leader_election.cc:288] T 00000000000000000000000000000000 P 12c2fb5fc9eb4fe2850ab1444aee8eda [CANDIDATE]: Term 2 election: Requested vote from peers 6d022764bb4a442e8466cb260239c7e5 (127.9.155.126:44831), c20a27597512484cb7546601dada9ca9 (127.9.155.125:35509)
I20260620 19:10:28.566556 17256 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c20a27597512484cb7546601dada9ca9 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:10:28.566812 17258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:28.566618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:44831 (local address 127.0.0.1:38906)
0620 19:10:28.566710 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:28.566757 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:44831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 19:10:28.566867 16554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "12c2fb5fc9eb4fe2850ab1444aee8eda" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c20a27597512484cb7546601dada9ca9"
W20260620 19:10:28.567044 16715 leader_election.cc:334] T 00000000000000000000000000000000 P 12c2fb5fc9eb4fe2850ab1444aee8eda [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6d022764bb4a442e8466cb260239c7e5 (127.9.155.126:44831): Network error: Client connection negotiation failed: client connection to 127.9.155.126:44831: connect: Connection refused (error 111)
I20260620 19:10:28.567623 17256 raft_consensus.cc:2726] T 00000000000000000000000000000000 P c20a27597512484cb7546601dada9ca9 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:10:28.568665 16554 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c20a27597512484cb7546601dada9ca9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12c2fb5fc9eb4fe2850ab1444aee8eda in term 2.
I20260620 19:10:28.568913 16713 leader_election.cc:302] T 00000000000000000000000000000000 P 12c2fb5fc9eb4fe2850ab1444aee8eda [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 12c2fb5fc9eb4fe2850ab1444aee8eda, c20a27597512484cb7546601dada9ca9; no voters: 6d022764bb4a442e8466cb260239c7e5
I20260620 19:10:28.569078 17255 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12c2fb5fc9eb4fe2850ab1444aee8eda [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:10:28.569274 17255 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12c2fb5fc9eb4fe2850ab1444aee8eda [term 2 LEADER]: Becoming Leader. State: Replica: 12c2fb5fc9eb4fe2850ab1444aee8eda, State: Running, Role: LEADER
I20260620 19:10:28.569511 17255 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12c2fb5fc9eb4fe2850ab1444aee8eda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6d022764bb4a442e8466cb260239c7e5" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 44831 } } peers { permanent_uuid: "c20a27597512484cb7546601dada9ca9" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 35509 } attrs { promote: false } } peers { permanent_uuid: "12c2fb5fc9eb4fe2850ab1444aee8eda" member_type: VOTER last_known_addr { host: "127.9.155.124" port: 46355 } attrs { promote: false } }
I20260620 19:10:28.570978 17265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12c2fb5fc9eb4fe2850ab1444aee8eda [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12c2fb5fc9eb4fe2850ab1444aee8eda. Latest consensus state: current_term: 2 leader_uuid: "12c2fb5fc9eb4fe2850ab1444aee8eda" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6d022764bb4a442e8466cb260239c7e5" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 44831 } } peers { permanent_uuid: "c20a27597512484cb7546601dada9ca9" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 35509 } attrs { promote: false } } peers { permanent_uuid: "12c2fb5fc9eb4fe2850ab1444aee8eda" member_type: VOTER last_known_addr { host: "127.9.155.124" port: 46355 } attrs { promote: false } } }
I20260620 19:10:28.571120 17265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12c2fb5fc9eb4fe2850ab1444aee8eda [sys.catalog]: This master's current role is: LEADER
I20260620 19:10:28.571420 17267 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:10:28.573773 17267 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:10:28.574154 17267 catalog_manager.cc:1259] Loaded cluster ID: 3e1c5ad4f96242d5b1943107518facd5
I20260620 19:10:28.574200 17267 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:10:28.574747 17267 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:10:28.575086 17267 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 12c2fb5fc9eb4fe2850ab1444aee8eda: Loaded TSK: 0
I20260620 19:10:28.575765 17267 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:10:28.603663 16729 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42990:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:10:28.606866 16729 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:10:28.609766 17258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:28.609543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:44831 (local address 127.0.0.1:38924)
0620 19:10:28.609648 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:28.609696 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:44831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 19:10:28.609887 16554 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c20a27597512484cb7546601dada9ca9 [term 2 FOLLOWER]: Refusing update from remote peer 12c2fb5fc9eb4fe2850ab1444aee8eda: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260620 19:10:28.610383 17265 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 12c2fb5fc9eb4fe2850ab1444aee8eda [LEADER]: Connected to new peer: Peer: permanent_uuid: "c20a27597512484cb7546601dada9ca9" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 35509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260620 19:10:28.610459 17258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:28.610237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:44831 (local address 127.0.0.1:38938)
0620 19:10:28.610339 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:28.610393 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:44831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260620 19:10:28.610702 16715 consensus_peers.cc:487] T 00000000000000000000000000000000 P 12c2fb5fc9eb4fe2850ab1444aee8eda -> Peer 6d022764bb4a442e8466cb260239c7e5 (127.9.155.126:44831): Couldn't send request to peer 6d022764bb4a442e8466cb260239c7e5. Status: Network error: Client connection negotiation failed: client connection to 127.9.155.126:44831: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 19:10:28.611915 17256 sys_catalog.cc:455] T 00000000000000000000000000000000 P c20a27597512484cb7546601dada9ca9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12c2fb5fc9eb4fe2850ab1444aee8eda. Latest consensus state: current_term: 2 leader_uuid: "12c2fb5fc9eb4fe2850ab1444aee8eda" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6d022764bb4a442e8466cb260239c7e5" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 44831 } } peers { permanent_uuid: "c20a27597512484cb7546601dada9ca9" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 35509 } attrs { promote: false } } peers { permanent_uuid: "12c2fb5fc9eb4fe2850ab1444aee8eda" member_type: VOTER last_known_addr { host: "127.9.155.124" port: 46355 } attrs { promote: false } } }
I20260620 19:10:28.612069 17256 sys_catalog.cc:458] T 00000000000000000000000000000000 P c20a27597512484cb7546601dada9ca9 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:10:28.613214 17255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12c2fb5fc9eb4fe2850ab1444aee8eda [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "12c2fb5fc9eb4fe2850ab1444aee8eda" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6d022764bb4a442e8466cb260239c7e5" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 44831 } } peers { permanent_uuid: "c20a27597512484cb7546601dada9ca9" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 35509 } attrs { promote: false } } peers { permanent_uuid: "12c2fb5fc9eb4fe2850ab1444aee8eda" member_type: VOTER last_known_addr { host: "127.9.155.124" port: 46355 } attrs { promote: false } } }
I20260620 19:10:28.613318 17255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12c2fb5fc9eb4fe2850ab1444aee8eda [sys.catalog]: This master's current role is: LEADER
I20260620 19:10:28.613675 17256 sys_catalog.cc:455] T 00000000000000000000000000000000 P c20a27597512484cb7546601dada9ca9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "12c2fb5fc9eb4fe2850ab1444aee8eda" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6d022764bb4a442e8466cb260239c7e5" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 44831 } } peers { permanent_uuid: "c20a27597512484cb7546601dada9ca9" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 35509 } attrs { promote: false } } peers { permanent_uuid: "12c2fb5fc9eb4fe2850ab1444aee8eda" member_type: VOTER last_known_addr { host: "127.9.155.124" port: 46355 } attrs { promote: false } } }
I20260620 19:10:28.613780 17256 sys_catalog.cc:458] T 00000000000000000000000000000000 P c20a27597512484cb7546601dada9ca9 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:10:28.614485 17269 mvcc.cc:205] Tried to move back new op lower bound from 7299000846781616128 to 7299000846623338496. Current Snapshot: MvccSnapshot[applied={T|T < 7299000846781616128 or (T in {7299000846781616128})}]
I20260620 19:10:28.623531 17258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:28.619063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.67:34277 (local address 127.0.0.1:42490)
0620 19:10:28.619174 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:28.619188 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:28.619263 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:28.619934 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:28.619937 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:28.619955 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:10:28.620081 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:28.620088 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:28.621427 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:28.621430 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:28.622223 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:28.622232 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:28.623033 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:28.623036 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:28.623217 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:28.623240 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 19:10:28.623302 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 19:10:28.623390 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":73}
I20260620 19:10:28.623659 17275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:28.619511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53122 (local address 127.9.155.66:44711)
0620 19:10:28.619706 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:28.619710 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:28.619723 (+ 13us) server_negotiation.cc:407] Connection header received
0620 19:10:28.619806 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:28.619809 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:28.619848 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:28.620081 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:10:28.620250 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:28.621001 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:28.622502 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:28.623137 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:28.623232 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:28.623297 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:28.623440 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 19:10:28.623526 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":62,"threads_started":1}
I20260620 19:10:28.623657 17257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:28.619298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.66:44711 (local address 127.0.0.1:53122)
0620 19:10:28.619641 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:28.619653 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:28.619727 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:28.619963 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:28.619966 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:28.619988 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:10:28.620156 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:28.620161 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:28.621127 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:28.621129 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:28.622357 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:28.622366 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:28.623242 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:28.623245 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:28.623330 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:28.623353 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 19:10:28.623417 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 19:10:28.623488 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"mutex_wait_us":120,"spinlock_wait_cycles":1280}
I20260620 19:10:28.624259 17274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:28.619422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42490 (local address 127.9.155.67:34277)
0620 19:10:28.619684 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:28.619689 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:28.619704 (+ 15us) server_negotiation.cc:407] Connection header received
0620 19:10:28.619755 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:28.619758 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:28.619805 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:28.620292 (+ 487us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:10:28.620691 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:28.621325 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:28.622361 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:28.622939 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:28.623037 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:28.623105 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:28.624009 (+ 904us) server_negotiation.cc:299] Negotiation successful
0620 19:10:28.624097 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":88,"threads_started":1}
I20260620 19:10:28.624253 16979 tablet_service.cc:1467] Processing CreateTablet for tablet a9510ae2b5674b95ac28df9d39b30abc (DEFAULT_TABLE table=test-table [id=1f11fc0cf5ca48b7aab06e91853c7d5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:10:28.624455 17276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:28.620052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.65:42443 (local address 127.0.0.1:50288)
0620 19:10:28.620316 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:28.620328 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:28.620404 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:28.620869 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:28.620872 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:28.620887 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:10:28.620980 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:28.620986 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:28.621958 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:28.621962 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:28.623227 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:28.623235 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:28.624068 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:28.624071 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:28.624159 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:28.624180 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 19:10:28.624268 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 19:10:28.624340 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":59,"thread_start_us":94,"threads_started":1}
I20260620 19:10:28.624565 16979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9510ae2b5674b95ac28df9d39b30abc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:28.624769 17117 tablet_service.cc:1467] Processing CreateTablet for tablet a9510ae2b5674b95ac28df9d39b30abc (DEFAULT_TABLE table=test-table [id=1f11fc0cf5ca48b7aab06e91853c7d5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:10:28.624878 17277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:28.620445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50288 (local address 127.9.155.65:42443)
0620 19:10:28.620673 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:28.620678 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:28.620691 (+ 13us) server_negotiation.cc:407] Connection header received
0620 19:10:28.620738 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:28.620741 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:28.620778 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:28.620851 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:10:28.621082 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:28.621836 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:28.623354 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:28.623947 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:28.624614 (+ 667us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:28.624680 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:28.624713 (+ 33us) server_negotiation.cc:299] Negotiation successful
0620 19:10:28.624772 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":80,"threads_started":1}
I20260620 19:10:28.625020 17117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9510ae2b5674b95ac28df9d39b30abc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:28.627216 17278 tablet_bootstrap.cc:492] T a9510ae2b5674b95ac28df9d39b30abc P 22fadda898e54ffd9a8eaac7a55fcb91: Bootstrap starting.
I20260620 19:10:28.627334 17279 tablet_bootstrap.cc:492] T a9510ae2b5674b95ac28df9d39b30abc P a2b48205cf584ab18dd060cf238791ab: Bootstrap starting.
I20260620 19:10:28.627425 16842 tablet_service.cc:1467] Processing CreateTablet for tablet a9510ae2b5674b95ac28df9d39b30abc (DEFAULT_TABLE table=test-table [id=1f11fc0cf5ca48b7aab06e91853c7d5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:10:28.627689 16842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9510ae2b5674b95ac28df9d39b30abc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:28.628178 17279 tablet_bootstrap.cc:654] T a9510ae2b5674b95ac28df9d39b30abc P a2b48205cf584ab18dd060cf238791ab: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:28.628178 17278 tablet_bootstrap.cc:654] T a9510ae2b5674b95ac28df9d39b30abc P 22fadda898e54ffd9a8eaac7a55fcb91: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:28.629415 17279 tablet_bootstrap.cc:492] T a9510ae2b5674b95ac28df9d39b30abc P a2b48205cf584ab18dd060cf238791ab: No bootstrap required, opened a new log
I20260620 19:10:28.629506 17279 ts_tablet_manager.cc:1397] T a9510ae2b5674b95ac28df9d39b30abc P a2b48205cf584ab18dd060cf238791ab: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 19:10:28.629766 17278 tablet_bootstrap.cc:492] T a9510ae2b5674b95ac28df9d39b30abc P 22fadda898e54ffd9a8eaac7a55fcb91: No bootstrap required, opened a new log
I20260620 19:10:28.629972 17278 ts_tablet_manager.cc:1397] T a9510ae2b5674b95ac28df9d39b30abc P 22fadda898e54ffd9a8eaac7a55fcb91: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260620 19:10:28.629993 17279 raft_consensus.cc:348] T a9510ae2b5674b95ac28df9d39b30abc P a2b48205cf584ab18dd060cf238791ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b48205cf584ab18dd060cf238791ab" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34277 } } peers { permanent_uuid: "22fadda898e54ffd9a8eaac7a55fcb91" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 44711 } } peers { permanent_uuid: "af15d4c4a9304cc8a20af15051256305" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 42443 } }
I20260620 19:10:28.630107 17279 raft_consensus.cc:374] T a9510ae2b5674b95ac28df9d39b30abc P a2b48205cf584ab18dd060cf238791ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:28.630146 17279 raft_consensus.cc:729] T a9510ae2b5674b95ac28df9d39b30abc P a2b48205cf584ab18dd060cf238791ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2b48205cf584ab18dd060cf238791ab, State: Initialized, Role: FOLLOWER
I20260620 19:10:28.630291 17279 consensus_queue.cc:260] T a9510ae2b5674b95ac28df9d39b30abc P a2b48205cf584ab18dd060cf238791ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b48205cf584ab18dd060cf238791ab" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34277 } } peers { permanent_uuid: "22fadda898e54ffd9a8eaac7a55fcb91" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 44711 } } peers { permanent_uuid: "af15d4c4a9304cc8a20af15051256305" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 42443 } }
I20260620 19:10:28.630419 17278 raft_consensus.cc:348] T a9510ae2b5674b95ac28df9d39b30abc P 22fadda898e54ffd9a8eaac7a55fcb91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b48205cf584ab18dd060cf238791ab" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34277 } } peers { permanent_uuid: "22fadda898e54ffd9a8eaac7a55fcb91" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 44711 } } peers { permanent_uuid: "af15d4c4a9304cc8a20af15051256305" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 42443 } }
I20260620 19:10:28.630517 17278 raft_consensus.cc:374] T a9510ae2b5674b95ac28df9d39b30abc P 22fadda898e54ffd9a8eaac7a55fcb91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:28.630546 17278 raft_consensus.cc:729] T a9510ae2b5674b95ac28df9d39b30abc P 22fadda898e54ffd9a8eaac7a55fcb91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22fadda898e54ffd9a8eaac7a55fcb91, State: Initialized, Role: FOLLOWER
I20260620 19:10:28.630628 17279 ts_tablet_manager.cc:1428] T a9510ae2b5674b95ac28df9d39b30abc P a2b48205cf584ab18dd060cf238791ab: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 19:10:28.630685 17278 consensus_queue.cc:260] T a9510ae2b5674b95ac28df9d39b30abc P 22fadda898e54ffd9a8eaac7a55fcb91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b48205cf584ab18dd060cf238791ab" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34277 } } peers { permanent_uuid: "22fadda898e54ffd9a8eaac7a55fcb91" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 44711 } } peers { permanent_uuid: "af15d4c4a9304cc8a20af15051256305" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 42443 } }
I20260620 19:10:28.630760 17282 tablet_bootstrap.cc:492] T a9510ae2b5674b95ac28df9d39b30abc P af15d4c4a9304cc8a20af15051256305: Bootstrap starting.
I20260620 19:10:28.630998 17278 ts_tablet_manager.cc:1428] T a9510ae2b5674b95ac28df9d39b30abc P 22fadda898e54ffd9a8eaac7a55fcb91: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 19:10:28.631650 17282 tablet_bootstrap.cc:654] T a9510ae2b5674b95ac28df9d39b30abc P af15d4c4a9304cc8a20af15051256305: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:28.632840 17282 tablet_bootstrap.cc:492] T a9510ae2b5674b95ac28df9d39b30abc P af15d4c4a9304cc8a20af15051256305: No bootstrap required, opened a new log
I20260620 19:10:28.632926 17282 ts_tablet_manager.cc:1397] T a9510ae2b5674b95ac28df9d39b30abc P af15d4c4a9304cc8a20af15051256305: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:10:28.633283 17282 raft_consensus.cc:348] T a9510ae2b5674b95ac28df9d39b30abc P af15d4c4a9304cc8a20af15051256305 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b48205cf584ab18dd060cf238791ab" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34277 } } peers { permanent_uuid: "22fadda898e54ffd9a8eaac7a55fcb91" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 44711 } } peers { permanent_uuid: "af15d4c4a9304cc8a20af15051256305" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 42443 } }
I20260620 19:10:28.633394 17282 raft_consensus.cc:374] T a9510ae2b5674b95ac28df9d39b30abc P af15d4c4a9304cc8a20af15051256305 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:28.633419 17287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:28.632609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:44831 (local address 127.9.155.66:44619)
0620 19:10:28.633288 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:28.633341 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:44831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":91,"threads_started":1}
I20260620 19:10:28.633425 17282 raft_consensus.cc:729] T a9510ae2b5674b95ac28df9d39b30abc P af15d4c4a9304cc8a20af15051256305 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af15d4c4a9304cc8a20af15051256305, State: Initialized, Role: FOLLOWER
I20260620 19:10:28.633543 17282 consensus_queue.cc:260] T a9510ae2b5674b95ac28df9d39b30abc P af15d4c4a9304cc8a20af15051256305 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b48205cf584ab18dd060cf238791ab" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34277 } } peers { permanent_uuid: "22fadda898e54ffd9a8eaac7a55fcb91" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 44711 } } peers { permanent_uuid: "af15d4c4a9304cc8a20af15051256305" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 42443 } }
I20260620 19:10:28.633836 17282 ts_tablet_manager.cc:1428] T a9510ae2b5674b95ac28df9d39b30abc P af15d4c4a9304cc8a20af15051256305: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:10:28.634264 17285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:28.631416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:44831 (local address 127.9.155.67:45843)
0620 19:10:28.634113 (+ 2697us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:28.634170 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:44831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2634,"thread_start_us":108,"threads_started":1}
I20260620 19:10:28.634502 17045 heartbeater.cc:503] Master 127.9.155.124:46355 was elected leader, sending a full tablet report...
I20260620 19:10:28.635268 17183 heartbeater.cc:503] Master 127.9.155.124:46355 was elected leader, sending a full tablet report...
I20260620 19:10:28.636162 16908 heartbeater.cc:503] Master 127.9.155.124:46355 was elected leader, sending a full tablet report...
I20260620 19:10:28.637095 17285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:28.636824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:44831 (local address 127.9.155.67:51079)
0620 19:10:28.636923 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:28.637025 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:44831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 19:10:28.638013 17287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:28.635002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:44831 (local address 127.9.155.66:44183)
0620 19:10:28.637670 (+ 2668us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:28.637944 (+ 274us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:44831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2600}
I20260620 19:10:28.638398 17289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:28.634282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:44831 (local address 127.9.155.65:34903)
0620 19:10:28.638260 (+ 3978us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:28.638316 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:44831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3919,"thread_start_us":82,"threads_started":1}
I20260620 19:10:28.639010 17289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:28.638794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:44831 (local address 127.9.155.65:59733)
0620 19:10:28.638894 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:28.638947 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:44831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 19:10:28.648578 17283 raft_consensus.cc:482] T a9510ae2b5674b95ac28df9d39b30abc P a2b48205cf584ab18dd060cf238791ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:10:28.648725 17283 raft_consensus.cc:504] T a9510ae2b5674b95ac28df9d39b30abc P a2b48205cf584ab18dd060cf238791ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b48205cf584ab18dd060cf238791ab" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34277 } } peers { permanent_uuid: "22fadda898e54ffd9a8eaac7a55fcb91" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 44711 } } peers { permanent_uuid: "af15d4c4a9304cc8a20af15051256305" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 42443 } }
I20260620 19:10:28.649840 17283 leader_election.cc:288] T a9510ae2b5674b95ac28df9d39b30abc P a2b48205cf584ab18dd060cf238791ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22fadda898e54ffd9a8eaac7a55fcb91 (127.9.155.66:44711), af15d4c4a9304cc8a20af15051256305 (127.9.155.65:42443)
I20260620 19:10:28.654675 17277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:28.650383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.67:43425 (local address 127.9.155.65:42443)
0620 19:10:28.650484 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:28.650488 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:28.650638 (+ 150us) server_negotiation.cc:407] Connection header received
0620 19:10:28.650746 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:28.650749 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:28.650795 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:28.650862 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:10:28.651051 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:28.651764 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:28.653436 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:28.654058 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:28.654195 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:28.654247 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:28.654503 (+ 256us) server_negotiation.cc:299] Negotiation successful
0620 19:10:28.654575 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 19:10:28.654673 17275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:28.650097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.67:41183 (local address 127.9.155.66:44711)
0620 19:10:28.650208 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:28.650211 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:28.650224 (+ 13us) server_negotiation.cc:407] Connection header received
0620 19:10:28.650485 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:28.650489 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:28.650539 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:28.650617 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:10:28.650895 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:28.651684 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:28.653436 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:28.654070 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:28.654195 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:28.654247 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:28.654464 (+ 217us) server_negotiation.cc:299] Negotiation successful
0620 19:10:28.654545 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 19:10:28.654686 17285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:28.650036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.66:44711 (local address 127.9.155.67:41183)
0620 19:10:28.650150 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:28.650165 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:28.650258 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:28.650640 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:28.650643 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:28.650666 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:10:28.650793 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:28.650800 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:28.651782 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:28.651785 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:28.653290 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:28.653303 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:28.654185 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:28.654188 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:28.654322 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:28.654372 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 19:10:28.654439 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 19:10:28.654521 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":112,"spinlock_wait_cycles":4224}
I20260620 19:10:28.654687 17290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:28.650286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.65:42443 (local address 127.9.155.67:43425)
0620 19:10:28.650557 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:28.650569 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:28.650667 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:28.650877 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:28.650879 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:28.650896 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:10:28.650963 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:28.650966 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:28.651861 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:28.651864 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:28.653290 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:28.653303 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:28.654185 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:28.654188 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:28.654325 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:28.654365 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 19:10:28.654439 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 19:10:28.654521 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":121,"spinlock_wait_cycles":6400,"thread_start_us":117,"threads_started":1}
I20260620 19:10:28.655339 16862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9510ae2b5674b95ac28df9d39b30abc" candidate_uuid: "a2b48205cf584ab18dd060cf238791ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af15d4c4a9304cc8a20af15051256305" is_pre_election: true
I20260620 19:10:28.655445 16999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9510ae2b5674b95ac28df9d39b30abc" candidate_uuid: "a2b48205cf584ab18dd060cf238791ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fadda898e54ffd9a8eaac7a55fcb91" is_pre_election: true
I20260620 19:10:28.655576 16999 raft_consensus.cc:2454] T a9510ae2b5674b95ac28df9d39b30abc P 22fadda898e54ffd9a8eaac7a55fcb91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2b48205cf584ab18dd060cf238791ab in term 0.
I20260620 19:10:28.655645 16862 raft_consensus.cc:2454] T a9510ae2b5674b95ac28df9d39b30abc P af15d4c4a9304cc8a20af15051256305 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2b48205cf584ab18dd060cf238791ab in term 0.
I20260620 19:10:28.655984 17071 leader_election.cc:302] T a9510ae2b5674b95ac28df9d39b30abc P a2b48205cf584ab18dd060cf238791ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fadda898e54ffd9a8eaac7a55fcb91, a2b48205cf584ab18dd060cf238791ab; no voters:
I20260620 19:10:28.656299 17283 raft_consensus.cc:2781] T a9510ae2b5674b95ac28df9d39b30abc P a2b48205cf584ab18dd060cf238791ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:10:28.656406 17283 raft_consensus.cc:482] T a9510ae2b5674b95ac28df9d39b30abc P a2b48205cf584ab18dd060cf238791ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:10:28.656462 17283 raft_consensus.cc:3037] T a9510ae2b5674b95ac28df9d39b30abc P a2b48205cf584ab18dd060cf238791ab [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:28.657747 17283 raft_consensus.cc:504] T a9510ae2b5674b95ac28df9d39b30abc P a2b48205cf584ab18dd060cf238791ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b48205cf584ab18dd060cf238791ab" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34277 } } peers { permanent_uuid: "22fadda898e54ffd9a8eaac7a55fcb91" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 44711 } } peers { permanent_uuid: "af15d4c4a9304cc8a20af15051256305" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 42443 } }
I20260620 19:10:28.658097 17283 leader_election.cc:288] T a9510ae2b5674b95ac28df9d39b30abc P a2b48205cf584ab18dd060cf238791ab [CANDIDATE]: Term 1 election: Requested vote from peers 22fadda898e54ffd9a8eaac7a55fcb91 (127.9.155.66:44711), af15d4c4a9304cc8a20af15051256305 (127.9.155.65:42443)
I20260620 19:10:28.658465 16999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9510ae2b5674b95ac28df9d39b30abc" candidate_uuid: "a2b48205cf584ab18dd060cf238791ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fadda898e54ffd9a8eaac7a55fcb91"
I20260620 19:10:28.658545 16862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9510ae2b5674b95ac28df9d39b30abc" candidate_uuid: "a2b48205cf584ab18dd060cf238791ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af15d4c4a9304cc8a20af15051256305"
I20260620 19:10:28.658586 16999 raft_consensus.cc:3037] T a9510ae2b5674b95ac28df9d39b30abc P 22fadda898e54ffd9a8eaac7a55fcb91 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:28.658671 16862 raft_consensus.cc:3037] T a9510ae2b5674b95ac28df9d39b30abc P af15d4c4a9304cc8a20af15051256305 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:28.659471 16999 raft_consensus.cc:2454] T a9510ae2b5674b95ac28df9d39b30abc P 22fadda898e54ffd9a8eaac7a55fcb91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2b48205cf584ab18dd060cf238791ab in term 1.
I20260620 19:10:28.659613 16862 raft_consensus.cc:2454] T a9510ae2b5674b95ac28df9d39b30abc P af15d4c4a9304cc8a20af15051256305 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2b48205cf584ab18dd060cf238791ab in term 1.
I20260620 19:10:28.659749 17071 leader_election.cc:302] T a9510ae2b5674b95ac28df9d39b30abc P a2b48205cf584ab18dd060cf238791ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fadda898e54ffd9a8eaac7a55fcb91, a2b48205cf584ab18dd060cf238791ab; no voters:
I20260620 19:10:28.659951 17283 raft_consensus.cc:2781] T a9510ae2b5674b95ac28df9d39b30abc P a2b48205cf584ab18dd060cf238791ab [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:10:28.660168 17283 raft_consensus.cc:686] T a9510ae2b5674b95ac28df9d39b30abc P a2b48205cf584ab18dd060cf238791ab [term 1 LEADER]: Becoming Leader. State: Replica: a2b48205cf584ab18dd060cf238791ab, State: Running, Role: LEADER
I20260620 19:10:28.660440 17283 consensus_queue.cc:237] T a9510ae2b5674b95ac28df9d39b30abc P a2b48205cf584ab18dd060cf238791ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b48205cf584ab18dd060cf238791ab" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34277 } } peers { permanent_uuid: "22fadda898e54ffd9a8eaac7a55fcb91" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 44711 } } peers { permanent_uuid: "af15d4c4a9304cc8a20af15051256305" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 42443 } }
I20260620 19:10:28.660864 17285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:28.660628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:44831 (local address 127.9.155.67:53539)
0620 19:10:28.660733 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:28.660791 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:44831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260620 19:10:28.662778 16727 catalog_manager.cc:5495] T a9510ae2b5674b95ac28df9d39b30abc P a2b48205cf584ab18dd060cf238791ab reported cstate change: term changed from 0 to 1, leader changed from <none> to a2b48205cf584ab18dd060cf238791ab (127.9.155.67). New cstate: current_term: 1 leader_uuid: "a2b48205cf584ab18dd060cf238791ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b48205cf584ab18dd060cf238791ab" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34277 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22fadda898e54ffd9a8eaac7a55fcb91" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 44711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af15d4c4a9304cc8a20af15051256305" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 42443 } health_report { overall_health: UNKNOWN } } }
I20260620 19:10:28.675551 17274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:28.672163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42496 (local address 127.9.155.67:34277)
0620 19:10:28.672309 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:28.672313 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:28.672329 (+ 16us) server_negotiation.cc:407] Connection header received
0620 19:10:28.672441 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:28.672444 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:28.672506 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:28.672613 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:10:28.672800 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:28.673295 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:28.674140 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:28.674348 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:28.674422 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:28.674704 (+ 282us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:10:28.675156 (+ 452us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:10:28.675240 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:28.675389 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 19:10:28.675439 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 19:10:28.678474 16999 raft_consensus.cc:1261] T a9510ae2b5674b95ac28df9d39b30abc P 22fadda898e54ffd9a8eaac7a55fcb91 [term 1 FOLLOWER]: Refusing update from remote peer a2b48205cf584ab18dd060cf238791ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:10:28.678728 16862 raft_consensus.cc:1261] T a9510ae2b5674b95ac28df9d39b30abc P af15d4c4a9304cc8a20af15051256305 [term 1 FOLLOWER]: Refusing update from remote peer a2b48205cf584ab18dd060cf238791ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:10:28.679024 17291 consensus_queue.cc:1035] T a9510ae2b5674b95ac28df9d39b30abc P a2b48205cf584ab18dd060cf238791ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fadda898e54ffd9a8eaac7a55fcb91" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 44711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:10:28.679241 17283 consensus_queue.cc:1035] T a9510ae2b5674b95ac28df9d39b30abc P a2b48205cf584ab18dd060cf238791ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "af15d4c4a9304cc8a20af15051256305" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 42443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:10:28.680895 17287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:28.680637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:44831 (local address 127.9.155.66:59127)
0620 19:10:28.680748 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:28.680813 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:44831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 19:10:28.682184 17289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:28.681335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:44831 (local address 127.9.155.65:48827)
0620 19:10:28.682057 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:28.682111 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:44831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":653}
I20260620 19:10:28.684407 17285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:28.684152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:44831 (local address 127.9.155.67:42663)
0620 19:10:28.684278 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:28.684333 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:44831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 19:10:28.686241 17287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:28.685998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:44831 (local address 127.9.155.66:34567)
0620 19:10:28.686108 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:28.686166 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:44831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 19:10:28.687213 17285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:28.686959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:44831 (local address 127.9.155.67:45893)
0620 19:10:28.687097 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:28.687147 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:44831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260620 19:10:28.687227 17289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:28.686646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:44831 (local address 127.9.155.65:33517)
0620 19:10:28.687122 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:28.687167 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:44831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415}
I20260620 19:10:28.692524 16729 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42990:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:10:28.692677 16729 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42990:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:10:28.696744 16729 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 12c2fb5fc9eb4fe2850ab1444aee8eda: Sending DeleteTablet for 3 replicas of tablet a9510ae2b5674b95ac28df9d39b30abc
I20260620 19:10:28.697438 17117 tablet_service.cc:1514] Processing DeleteTablet for tablet a9510ae2b5674b95ac28df9d39b30abc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:10:28 UTC) from {username='oryx', principal='oryx/127.9.155.124@KRBTEST.COM'} at 127.0.0.1:42490
I20260620 19:10:28.697602 16842 tablet_service.cc:1514] Processing DeleteTablet for tablet a9510ae2b5674b95ac28df9d39b30abc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:10:28 UTC) from {username='oryx', principal='oryx/127.9.155.124@KRBTEST.COM'} at 127.0.0.1:50288
I20260620 19:10:28.697851 17303 tablet_replica.cc:330] T a9510ae2b5674b95ac28df9d39b30abc P a2b48205cf584ab18dd060cf238791ab: stopping tablet replica
I20260620 19:10:28.697917 16979 tablet_service.cc:1514] Processing DeleteTablet for tablet a9510ae2b5674b95ac28df9d39b30abc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:10:28 UTC) from {username='oryx', principal='oryx/127.9.155.124@KRBTEST.COM'} at 127.0.0.1:53122
I20260620 19:10:28.697990 17303 raft_consensus.cc:2229] T a9510ae2b5674b95ac28df9d39b30abc P a2b48205cf584ab18dd060cf238791ab [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:10:28.698323 17305 tablet_replica.cc:330] T a9510ae2b5674b95ac28df9d39b30abc P 22fadda898e54ffd9a8eaac7a55fcb91: stopping tablet replica
I20260620 19:10:28.698338 17303 raft_consensus.cc:2258] T a9510ae2b5674b95ac28df9d39b30abc P a2b48205cf584ab18dd060cf238791ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:10:28.698541 17305 raft_consensus.cc:2229] T a9510ae2b5674b95ac28df9d39b30abc P 22fadda898e54ffd9a8eaac7a55fcb91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:10:28.698730 17305 raft_consensus.cc:2258] T a9510ae2b5674b95ac28df9d39b30abc P 22fadda898e54ffd9a8eaac7a55fcb91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:10:28.699141 17304 tablet_replica.cc:330] T a9510ae2b5674b95ac28df9d39b30abc P af15d4c4a9304cc8a20af15051256305: stopping tablet replica
I20260620 19:10:28.699321 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 16688
I20260620 19:10:28.705430 17303 ts_tablet_manager.cc:1905] T a9510ae2b5674b95ac28df9d39b30abc P a2b48205cf584ab18dd060cf238791ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:10:28.706677 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 16916
I20260620 19:10:28.708284 17303 ts_tablet_manager.cc:1918] T a9510ae2b5674b95ac28df9d39b30abc P a2b48205cf584ab18dd060cf238791ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:10:28.708369 17303 log.cc:1196] T a9510ae2b5674b95ac28df9d39b30abc P a2b48205cf584ab18dd060cf238791ab: Deleting WAL directory at /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982595231230-9837-0/minicluster-data/ts-2/wal/wals/a9510ae2b5674b95ac28df9d39b30abc
I20260620 19:10:28.708412 17276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:28.708052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.65:42443 (local address 127.0.0.1:50296)
0620 19:10:28.708265 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:28.708320 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.65:42443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260620 19:10:28.708683 17303 ts_tablet_manager.cc:1939] T a9510ae2b5674b95ac28df9d39b30abc P a2b48205cf584ab18dd060cf238791ab: Deleting consensus metadata
W20260620 19:10:28.708830 16714 catalog_manager.cc:4584] TS af15d4c4a9304cc8a20af15051256305 (127.9.155.65:42443): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a9510ae2b5674b95ac28df9d39b30abc: Network error: Client connection negotiation failed: client connection to 127.9.155.65:42443: connect: Connection refused (error 111)
I20260620 19:10:28.709422 16714 catalog_manager.cc:4841] TS a2b48205cf584ab18dd060cf238791ab (127.9.155.67:34277): tablet a9510ae2b5674b95ac28df9d39b30abc (table test-table [id=1f11fc0cf5ca48b7aab06e91853c7d5b]) successfully deleted
I20260620 19:10:28.712850 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 17052
I20260620 19:10:28.713085 17276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:28.712826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.66:44711 (local address 127.0.0.1:53130)
0620 19:10:28.712943 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:28.713007 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.66:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 19:10:28.717563 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 16407
I20260620 19:10:28.722671 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 16516
2026-06-20T19:10:28Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5174 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:10:28 dist-test-slave-n3fx krb5kdc[17308](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:10:28 dist-test-slave-n3fx krb5kdc[17308](info): set up 2 sockets
Jun 20 19:10:28 dist-test-slave-n3fx krb5kdc[17308](info): commencing operation
krb5kdc: starting...
W20260620 19:10:29.692482 9837 mini_kdc.cc:121] Time spent starting KDC: real 0.948s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/krb5kdc/test-user.keytab.
Jun 20 19:10:29 dist-test-slave-n3fx krb5kdc[17308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982629, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:10:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:10:29Z Disabled control of system clock
WARNING: no policy specified for oryx/127.9.155.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.155.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.155.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/krb5kdc/oryx_127.9.155.126.keytab.
Entry for principal oryx/127.9.155.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/krb5kdc/oryx_127.9.155.126.keytab.
I20260620 19:10:29.806870 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:39669
--webserver_interface=127.9.155.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/krb5kdc/oryx_127.9.155.126.keytab
--principal=oryx/127.9.155.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:42983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.155.126:39669
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:10:29.915958 17321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:29.916311 17321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:29.916373 17321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:29.920539 17321 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:10:29.920614 17321 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 19:10:29.920639 17321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:29.920661 17321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:29.920684 17321 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:10:29.920710 17321 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:10:29.926323 17321 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:42983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.155.126:39669
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/krb5kdc/oryx_127.9.155.126.keytab
--principal=oryx/127.9.155.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:39669
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.155.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17321
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:10:29.927450 17321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:29.928575 17321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:10:29.937263 17327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:29.937362 17321 server_base.cc:1034] running on GCE node
W20260620 19:10:29.937256 17326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:29.937263 17329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:29.938145 17321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:29.939284 17321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:29.940449 17321 hybrid_clock.cc:648] HybridClock initialized: now 1781982629940435 us; error 30 us; skew 500 ppm
Jun 20 19:10:29 dist-test-slave-n3fx krb5kdc[17308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982629, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.155.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:29.946802 17321 init.cc:377] Logged in from keytab as oryx/127.9.155.126@KRBTEST.COM (short username oryx)
I20260620 19:10:29.947824 17321 webserver.cc:460] Webserver started at http://127.9.155.126:40677/ using document root <none> and password file <none>
I20260620 19:10:29.948451 17321 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:29.948518 17321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:29.948719 17321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:10:29.950881 17321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/minicluster-data/master-0/data/instance:
uuid: "42920bad573b462593296d6a1276cb32"
format_stamp: "Formatted at 2026-06-20 19:10:29 on dist-test-slave-n3fx"
server_key: "e8fb02cfaf8b0b70ae6aab0b2e92a46b"
server_key_iv: "6d1143f989c4ed75a3dd1f600de781c1"
server_key_version: "encryptionkey@0"
I20260620 19:10:29.951481 17321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/minicluster-data/master-0/wal/instance:
uuid: "42920bad573b462593296d6a1276cb32"
format_stamp: "Formatted at 2026-06-20 19:10:29 on dist-test-slave-n3fx"
server_key: "e8fb02cfaf8b0b70ae6aab0b2e92a46b"
server_key_iv: "6d1143f989c4ed75a3dd1f600de781c1"
server_key_version: "encryptionkey@0"
I20260620 19:10:29.956164 17321 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260620 19:10:29.959084 17336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:29.960812 17321 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:10:29.960960 17321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/minicluster-data/master-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/minicluster-data/master-0/wal
uuid: "42920bad573b462593296d6a1276cb32"
format_stamp: "Formatted at 2026-06-20 19:10:29 on dist-test-slave-n3fx"
server_key: "e8fb02cfaf8b0b70ae6aab0b2e92a46b"
server_key_iv: "6d1143f989c4ed75a3dd1f600de781c1"
server_key_version: "encryptionkey@0"
I20260620 19:10:29.961108 17321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:29.980788 17321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:29.984105 17321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:29.984326 17321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:29.992015 17321 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.126:39669
I20260620 19:10:29.992063 17398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.126:39669 every 8 connection(s)
I20260620 19:10:29.993042 17321 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
I20260620 19:10:29.993196 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 17321
I20260620 19:10:29.993319 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/minicluster-data/master-0/wal/instance
I20260620 19:10:29.993637 9837 external_mini_cluster.cc:1428] Setting key c2d128e585a1215a8440812104b88e41
I20260620 19:10:29.996464 17399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 19:10:29 dist-test-slave-n3fx krb5kdc[17308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982629, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM
I20260620 19:10:30.003230 17399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42920bad573b462593296d6a1276cb32: Bootstrap starting.
I20260620 19:10:30.009117 17399 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42920bad573b462593296d6a1276cb32: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:30.010574 17399 log.cc:824] T 00000000000000000000000000000000 P 42920bad573b462593296d6a1276cb32: Log is configured to *not* fsync() on all Append() calls
I20260620 19:10:30.011420 17402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:29.995065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52082 (local address 127.9.155.126:39669)
0620 19:10:29.995444 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:29.995460 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:29.995504 (+ 44us) server_negotiation.cc:407] Connection header received
0620 19:10:29.996187 (+ 683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:29.996205 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:29.996627 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:29.996927 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:29.997772 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:29.999109 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:30.000309 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:30.000806 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:30.000936 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:30.003791 (+ 2855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:30.003797 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:30.003811 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:30.003847 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:30.006814 (+ 2967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:30.007381 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:30.007387 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:30.007392 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:30.007518 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:30.007800 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:30.007803 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:30.007805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:30.009412 (+ 1607us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:30.009629 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:30.009991 (+ 362us) server_negotiation.cc:299] Negotiation successful
0620 19:10:30.010559 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":99,"threads_started":1}
I20260620 19:10:30.013731 17399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42920bad573b462593296d6a1276cb32: No bootstrap required, opened a new log
I20260620 19:10:30.017236 17399 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42920bad573b462593296d6a1276cb32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42920bad573b462593296d6a1276cb32" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 39669 } }
I20260620 19:10:30.017539 17399 raft_consensus.cc:374] T 00000000000000000000000000000000 P 42920bad573b462593296d6a1276cb32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:30.017589 17399 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42920bad573b462593296d6a1276cb32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42920bad573b462593296d6a1276cb32, State: Initialized, Role: FOLLOWER
I20260620 19:10:30.018066 17399 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42920bad573b462593296d6a1276cb32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42920bad573b462593296d6a1276cb32" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 39669 } }
I20260620 19:10:30.018215 17399 raft_consensus.cc:388] T 00000000000000000000000000000000 P 42920bad573b462593296d6a1276cb32 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:10:30.018267 17399 raft_consensus.cc:482] T 00000000000000000000000000000000 P 42920bad573b462593296d6a1276cb32 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:10:30.018373 17399 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42920bad573b462593296d6a1276cb32 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:30.019425 17399 raft_consensus.cc:504] T 00000000000000000000000000000000 P 42920bad573b462593296d6a1276cb32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42920bad573b462593296d6a1276cb32" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 39669 } }
I20260620 19:10:30.019788 17399 leader_election.cc:302] T 00000000000000000000000000000000 P 42920bad573b462593296d6a1276cb32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 42920bad573b462593296d6a1276cb32; no voters:
I20260620 19:10:30.020128 17399 leader_election.cc:288] T 00000000000000000000000000000000 P 42920bad573b462593296d6a1276cb32 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:10:30.020327 17404 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 42920bad573b462593296d6a1276cb32 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:10:30.020539 17404 raft_consensus.cc:686] T 00000000000000000000000000000000 P 42920bad573b462593296d6a1276cb32 [term 1 LEADER]: Becoming Leader. State: Replica: 42920bad573b462593296d6a1276cb32, State: Running, Role: LEADER
I20260620 19:10:30.020864 17404 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42920bad573b462593296d6a1276cb32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42920bad573b462593296d6a1276cb32" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 39669 } }
I20260620 19:10:30.021689 17399 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42920bad573b462593296d6a1276cb32 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:10:30.022890 17405 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42920bad573b462593296d6a1276cb32 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "42920bad573b462593296d6a1276cb32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42920bad573b462593296d6a1276cb32" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 39669 } } }
I20260620 19:10:30.022902 17406 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42920bad573b462593296d6a1276cb32 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42920bad573b462593296d6a1276cb32. Latest consensus state: current_term: 1 leader_uuid: "42920bad573b462593296d6a1276cb32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42920bad573b462593296d6a1276cb32" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 39669 } } }
I20260620 19:10:30.023161 17405 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42920bad573b462593296d6a1276cb32 [sys.catalog]: This master's current role is: LEADER
I20260620 19:10:30.023188 17406 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42920bad573b462593296d6a1276cb32 [sys.catalog]: This master's current role is: LEADER
I20260620 19:10:30.024112 17415 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:10:30.026815 17415 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:10:30.033149 17415 catalog_manager.cc:1347] Generated new cluster ID: 1a06197fcd884683bb34a04f5d7c71f9
I20260620 19:10:30.033242 17415 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:10:30.047928 17415 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:10:30.049208 17415 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:10:30.059985 17415 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 42920bad573b462593296d6a1276cb32: Generated new TSK 0
I20260620 19:10:30.060792 17415 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.9.155.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.155.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.155.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/krb5kdc/oryx_127.9.155.65.keytab.
Entry for principal oryx/127.9.155.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/krb5kdc/oryx_127.9.155.65.keytab.
I20260620 19:10:30.092285 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.65:0
--local_ip_for_outbound_sockets=127.9.155.65
--webserver_interface=127.9.155.65
--webserver_port=0
--tserver_master_addrs=127.9.155.126:39669
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/krb5kdc/oryx_127.9.155.65.keytab
--principal=oryx/127.9.155.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:42983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:10:30.206626 17425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:30.206933 17425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:30.206997 17425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:30.211277 17425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:30.211369 17425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:30.211422 17425 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 19:10:30.211479 17425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.65
I20260620 19:10:30.217090 17425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:42983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/krb5kdc/oryx_127.9.155.65.keytab
--principal=oryx/127.9.155.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.155.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.155.126:39669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.65
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:10:30.218272 17425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:30.219307 17425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:10:30.228972 17431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:30.229017 17433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:30.229005 17430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:30.229275 17425 server_base.cc:1034] running on GCE node
I20260620 19:10:30.229671 17425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:30.230284 17425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:30.231458 17425 hybrid_clock.cc:648] HybridClock initialized: now 1781982630231435 us; error 40 us; skew 500 ppm
Jun 20 19:10:30 dist-test-slave-n3fx krb5kdc[17308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982630, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.155.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:30.234073 17425 init.cc:377] Logged in from keytab as oryx/127.9.155.65@KRBTEST.COM (short username oryx)
I20260620 19:10:30.234954 17425 webserver.cc:460] Webserver started at http://127.9.155.65:40447/ using document root <none> and password file <none>
I20260620 19:10:30.235483 17425 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:30.235558 17425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:30.235761 17425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:10:30.237769 17425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/minicluster-data/ts-0/data/instance:
uuid: "35a74422fbe4496abc46cf71d22af858"
format_stamp: "Formatted at 2026-06-20 19:10:30 on dist-test-slave-n3fx"
server_key: "8071e504e21e1450fd5a5b1e24e016ae"
server_key_iv: "f28eca1cd740b368a109805f27f5f794"
server_key_version: "encryptionkey@0"
I20260620 19:10:30.238296 17425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/minicluster-data/ts-0/wal/instance:
uuid: "35a74422fbe4496abc46cf71d22af858"
format_stamp: "Formatted at 2026-06-20 19:10:30 on dist-test-slave-n3fx"
server_key: "8071e504e21e1450fd5a5b1e24e016ae"
server_key_iv: "f28eca1cd740b368a109805f27f5f794"
server_key_version: "encryptionkey@0"
I20260620 19:10:30.242089 17425 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 19:10:30.244846 17440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:30.246174 17425 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:10:30.246304 17425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/minicluster-data/ts-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/minicluster-data/ts-0/wal
uuid: "35a74422fbe4496abc46cf71d22af858"
format_stamp: "Formatted at 2026-06-20 19:10:30 on dist-test-slave-n3fx"
server_key: "8071e504e21e1450fd5a5b1e24e016ae"
server_key_iv: "f28eca1cd740b368a109805f27f5f794"
server_key_version: "encryptionkey@0"
I20260620 19:10:30.246416 17425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:30.267578 17425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:30.270684 17425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:30.270869 17425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:30.272372 17425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:10:30.272424 17425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:30.272478 17425 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:10:30.272506 17425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 19:10:30 dist-test-slave-n3fx krb5kdc[17308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982630, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.155.65@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM
I20260620 19:10:30.282213 17425 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.65:45895
I20260620 19:10:30.282414 17555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.65:45895 every 8 connection(s)
I20260620 19:10:30.283179 17425 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
I20260620 19:10:30.285537 17402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:30.273143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.65:46363 (local address 127.9.155.126:39669)
0620 19:10:30.273283 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:30.273288 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:30.274152 (+ 864us) server_negotiation.cc:407] Connection header received
0620 19:10:30.275131 (+ 979us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:30.275135 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:30.275191 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:30.275291 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:30.276552 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:30.277440 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:30.278713 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:30.278971 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:30.279072 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:30.281802 (+ 2730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:30.281806 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:30.281809 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:30.281836 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:30.283207 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:30.283800 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:30.283804 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:30.283805 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:30.283898 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:30.284461 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:30.284464 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:30.284466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:30.284661 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:30.284747 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:30.285241 (+ 494us) server_negotiation.cc:299] Negotiation successful
0620 19:10:30.285413 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 19:10:30.286240 17449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:30.273549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:39669 (local address 127.9.155.65:46363)
0620 19:10:30.273991 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:30.274040 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:30.274920 (+ 880us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:30.275467 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:30.275478 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:30.275822 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:30.276401 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:30.276421 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:30.277561 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:30.277564 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:30.278599 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:30.278609 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:30.279136 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:30.279141 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:30.279319 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:30.279914 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:30.279931 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:30.281631 (+ 1700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:30.283344 (+ 1713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:30.283350 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:30.283364 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:30.283692 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:30.284143 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:30.284145 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:30.284146 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:30.284330 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:30.284758 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:30.284763 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:30.285002 (+ 239us) client_negotiation.cc:769] Sending connection context
0620 19:10:30.285149 (+ 147us) client_negotiation.cc:240] Negotiation successful
0620 19:10:30.285460 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":97,"threads_started":1}
I20260620 19:10:30.287454 17556 heartbeater.cc:347] Connected to a master server at 127.9.155.126:39669
I20260620 19:10:30.287611 17556 heartbeater.cc:464] Registering TS with master...
I20260620 19:10:30.288146 17556 heartbeater.cc:511] Master 127.9.155.126:39669 requested a full tablet report, sending...
I20260620 19:10:30.289552 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 17425
I20260620 19:10:30.289675 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/minicluster-data/ts-0/wal/instance
I20260620 19:10:30.289702 17352 ts_manager.cc:194] Registered new tserver with Master: 35a74422fbe4496abc46cf71d22af858 (127.9.155.65:45895)
I20260620 19:10:30.289963 9837 external_mini_cluster.cc:1428] Setting key aa5bcf2ec8343e7ad77071340eca3c84
I20260620 19:10:30.291172 17352 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.155.65@KRBTEST.COM'} at 127.9.155.65:46363
I20260620 19:10:30.297863 17402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:30.289712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.65:48219 (local address 127.9.155.126:39669)
0620 19:10:30.289818 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:30.289823 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:30.290451 (+ 628us) server_negotiation.cc:407] Connection header received
0620 19:10:30.290645 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:30.290648 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:30.290686 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:30.290750 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:30.291403 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:30.292053 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:30.293149 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:30.293371 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:30.293456 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:30.294721 (+ 1265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:30.294725 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:30.294726 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:30.294749 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:30.296160 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:30.296644 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:30.296646 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:30.296647 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:30.296727 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:30.297072 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:30.297075 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:30.297076 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:30.297259 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:30.297342 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:30.297639 (+ 297us) server_negotiation.cc:299] Negotiation successful
0620 19:10:30.297746 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 19:10:30.298041 17562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:30.289629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:39669 (local address 127.9.155.65:48219)
0620 19:10:30.290375 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:30.290388 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:30.290514 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:30.290777 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:30.290780 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:30.290957 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:30.291054 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:30.291059 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:30.292174 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:30.292177 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:30.293050 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:30.293062 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:30.293466 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:30.293470 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:30.293561 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:30.294122 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:30.294140 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:30.294586 (+ 446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:30.296300 (+ 1714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:30.296303 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:30.296306 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:30.296552 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:30.296809 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:30.296812 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:30.296814 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:30.296922 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:30.297353 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:30.297356 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:30.297459 (+ 103us) client_negotiation.cc:769] Sending connection context
0620 19:10:30.297575 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 19:10:30.297723 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":667,"mutex_wait_us":2,"thread_start_us":92,"threads_started":1}
WARNING: no policy specified for oryx/127.9.155.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.155.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.155.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/krb5kdc/oryx_127.9.155.66.keytab.
Entry for principal oryx/127.9.155.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/krb5kdc/oryx_127.9.155.66.keytab.
I20260620 19:10:30.316151 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.66:0
--local_ip_for_outbound_sockets=127.9.155.66
--webserver_interface=127.9.155.66
--webserver_port=0
--tserver_master_addrs=127.9.155.126:39669
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/krb5kdc/oryx_127.9.155.66.keytab
--principal=oryx/127.9.155.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:42983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:10:30.429302 17565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:30.429579 17565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:30.429643 17565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:30.433704 17565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:30.433770 17565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:30.433826 17565 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 19:10:30.433893 17565 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.66
I20260620 19:10:30.439528 17565 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:42983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/krb5kdc/oryx_127.9.155.66.keytab
--principal=oryx/127.9.155.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.155.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.155.126:39669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17565
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.66
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:10:30.440655 17565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:30.441655 17565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:10:30.451169 17571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:30.451179 17573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:30.451179 17570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:30.451390 17565 server_base.cc:1034] running on GCE node
I20260620 19:10:30.451809 17565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:30.452519 17565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:30.453682 17565 hybrid_clock.cc:648] HybridClock initialized: now 1781982630453661 us; error 37 us; skew 500 ppm
Jun 20 19:10:30 dist-test-slave-n3fx krb5kdc[17308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982630, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.155.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:30.456491 17565 init.cc:377] Logged in from keytab as oryx/127.9.155.66@KRBTEST.COM (short username oryx)
I20260620 19:10:30.457455 17565 webserver.cc:460] Webserver started at http://127.9.155.66:44453/ using document root <none> and password file <none>
I20260620 19:10:30.458027 17565 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:30.458101 17565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:30.458405 17565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:10:30.460791 17565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/minicluster-data/ts-1/data/instance:
uuid: "b22f4448136b4b06945ad3ffa8b0231a"
format_stamp: "Formatted at 2026-06-20 19:10:30 on dist-test-slave-n3fx"
server_key: "cbd5ab661d6dd0912e407e6a3e1d135b"
server_key_iv: "2a4a3b818d456c82e4a705fa005ff4b4"
server_key_version: "encryptionkey@0"
I20260620 19:10:30.461381 17565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/minicluster-data/ts-1/wal/instance:
uuid: "b22f4448136b4b06945ad3ffa8b0231a"
format_stamp: "Formatted at 2026-06-20 19:10:30 on dist-test-slave-n3fx"
server_key: "cbd5ab661d6dd0912e407e6a3e1d135b"
server_key_iv: "2a4a3b818d456c82e4a705fa005ff4b4"
server_key_version: "encryptionkey@0"
I20260620 19:10:30.465562 17565 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 19:10:30.468127 17580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:30.469398 17565 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:10:30.469524 17565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/minicluster-data/ts-1/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/minicluster-data/ts-1/wal
uuid: "b22f4448136b4b06945ad3ffa8b0231a"
format_stamp: "Formatted at 2026-06-20 19:10:30 on dist-test-slave-n3fx"
server_key: "cbd5ab661d6dd0912e407e6a3e1d135b"
server_key_iv: "2a4a3b818d456c82e4a705fa005ff4b4"
server_key_version: "encryptionkey@0"
I20260620 19:10:30.469627 17565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:30.493475 17565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:30.496860 17565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:30.497040 17565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:30.498556 17565 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:10:30.498612 17565 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:30.498667 17565 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:10:30.498692 17565 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 19:10:30 dist-test-slave-n3fx krb5kdc[17308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982630, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.155.66@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM
I20260620 19:10:30.509013 17565 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.66:39779
I20260620 19:10:30.509704 17695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.66:39779 every 8 connection(s)
I20260620 19:10:30.510061 17565 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
I20260620 19:10:30.511590 17402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:30.499338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.66:36213 (local address 127.9.155.126:39669)
0620 19:10:30.499446 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:30.499449 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:30.500592 (+ 1143us) server_negotiation.cc:407] Connection header received
0620 19:10:30.501603 (+ 1011us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:30.501607 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:30.501670 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:30.501770 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:30.503037 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:30.503848 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:30.505118 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:30.505454 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:30.505572 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:30.508017 (+ 2445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:30.508026 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:30.508029 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:30.508058 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:30.509466 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:30.510010 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:30.510014 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:30.510015 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:30.510097 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:30.510392 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:30.510395 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:30.510397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:30.510623 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:30.510711 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:30.511350 (+ 639us) server_negotiation.cc:299] Negotiation successful
0620 19:10:30.511460 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 19:10:30.512584 17589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:30.499912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:39669 (local address 127.9.155.66:36213)
0620 19:10:30.500450 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:30.500493 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:30.501336 (+ 843us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:30.501943 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:30.501950 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:30.502328 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:30.502847 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:30.502865 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:30.503974 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:30.503978 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:30.505014 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:30.505025 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:30.505615 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:30.505619 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:30.505807 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:30.506438 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:30.506453 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:30.507849 (+ 1396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:30.509570 (+ 1721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:30.509575 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:30.509590 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:30.509913 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:30.510177 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:30.510180 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:30.510182 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:30.510312 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:30.510887 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:30.510893 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:30.511119 (+ 226us) client_negotiation.cc:769] Sending connection context
0620 19:10:30.511293 (+ 174us) client_negotiation.cc:240] Negotiation successful
0620 19:10:30.511682 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":134,"threads_started":1}
I20260620 19:10:30.513463 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 17565
I20260620 19:10:30.513572 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/minicluster-data/ts-1/wal/instance
I20260620 19:10:30.513816 17696 heartbeater.cc:347] Connected to a master server at 127.9.155.126:39669
I20260620 19:10:30.513875 9837 external_mini_cluster.cc:1428] Setting key e1ff814c3747fabb046a544014373971
I20260620 19:10:30.513957 17696 heartbeater.cc:464] Registering TS with master...
I20260620 19:10:30.514493 17696 heartbeater.cc:511] Master 127.9.155.126:39669 requested a full tablet report, sending...
I20260620 19:10:30.515515 17353 ts_manager.cc:194] Registered new tserver with Master: b22f4448136b4b06945ad3ffa8b0231a (127.9.155.66:39779)
I20260620 19:10:30.516114 17353 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.155.66@KRBTEST.COM'} at 127.9.155.66:36213
WARNING: no policy specified for oryx/127.9.155.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.155.67@KRBTEST.COM" created.
I20260620 19:10:30.524859 17402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:30.517058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.66:34647 (local address 127.9.155.126:39669)
0620 19:10:30.517166 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:30.517170 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:30.517649 (+ 479us) server_negotiation.cc:407] Connection header received
0620 19:10:30.517751 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:30.517754 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:30.517799 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:30.517874 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:30.518358 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:30.518856 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:30.519648 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:30.519869 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:30.519955 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:30.521153 (+ 1198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:30.521157 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:30.521160 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:30.521184 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:30.522926 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:30.523340 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:30.523343 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:30.523345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:30.523426 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:30.523848 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:30.523851 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:30.523853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:30.524106 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:30.524193 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:30.524585 (+ 392us) server_negotiation.cc:299] Negotiation successful
0620 19:10:30.524687 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 19:10:30.524874 17703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:30.517265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:39669 (local address 127.9.155.66:34647)
0620 19:10:30.517550 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:30.517565 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:30.517672 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:30.517913 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:30.517916 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:30.518142 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:30.518246 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:30.518253 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:30.518975 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:30.518979 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:30.519541 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:30.519549 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:30.519980 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:30.519983 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:30.520074 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:30.520669 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:30.520683 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:30.521018 (+ 335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:30.523017 (+ 1999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:30.523021 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:30.523023 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:30.523257 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:30.523647 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:30.523649 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:30.523650 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:30.523745 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:30.524195 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:30.524198 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:30.524382 (+ 184us) client_negotiation.cc:769] Sending connection context
0620 19:10:30.524496 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 19:10:30.524695 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":101,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.155.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/krb5kdc/oryx_127.9.155.67.keytab.
Entry for principal oryx/127.9.155.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/krb5kdc/oryx_127.9.155.67.keytab.
I20260620 19:10:30.540555 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.67:0
--local_ip_for_outbound_sockets=127.9.155.67
--webserver_interface=127.9.155.67
--webserver_port=0
--tserver_master_addrs=127.9.155.126:39669
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/krb5kdc/oryx_127.9.155.67.keytab
--principal=oryx/127.9.155.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:42983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:10:30.652467 17705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:30.652755 17705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:30.652803 17705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:30.656934 17705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:30.657007 17705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:30.657056 17705 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 19:10:30.657114 17705 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.67
I20260620 19:10:30.662485 17705 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:42983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/krb5kdc/oryx_127.9.155.67.keytab
--principal=oryx/127.9.155.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.155.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.155.126:39669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17705
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.67
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:10:30.663583 17705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:30.664655 17705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:10:30.674420 17711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:30.674430 17710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:30.674430 17713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:30.674669 17705 server_base.cc:1034] running on GCE node
I20260620 19:10:30.675069 17705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:30.675709 17705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:30.676853 17705 hybrid_clock.cc:648] HybridClock initialized: now 1781982630676828 us; error 38 us; skew 500 ppm
Jun 20 19:10:30 dist-test-slave-n3fx krb5kdc[17308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982630, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.155.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:30.679471 17705 init.cc:377] Logged in from keytab as oryx/127.9.155.67@KRBTEST.COM (short username oryx)
I20260620 19:10:30.680434 17705 webserver.cc:460] Webserver started at http://127.9.155.67:34299/ using document root <none> and password file <none>
I20260620 19:10:30.680985 17705 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:30.681049 17705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:30.681241 17705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:10:30.683367 17705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/minicluster-data/ts-2/data/instance:
uuid: "3c96b775eb0f489099f6169dd85b2683"
format_stamp: "Formatted at 2026-06-20 19:10:30 on dist-test-slave-n3fx"
server_key: "d298a744a28de114b2aab7fe73de1800"
server_key_iv: "0b0da11eca08a6f5b9f08a82279d97c9"
server_key_version: "encryptionkey@0"
I20260620 19:10:30.683947 17705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/minicluster-data/ts-2/wal/instance:
uuid: "3c96b775eb0f489099f6169dd85b2683"
format_stamp: "Formatted at 2026-06-20 19:10:30 on dist-test-slave-n3fx"
server_key: "d298a744a28de114b2aab7fe73de1800"
server_key_iv: "0b0da11eca08a6f5b9f08a82279d97c9"
server_key_version: "encryptionkey@0"
I20260620 19:10:30.687813 17705 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 19:10:30.690472 17720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:30.691838 17705 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 19:10:30.691946 17705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/minicluster-data/ts-2/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/minicluster-data/ts-2/wal
uuid: "3c96b775eb0f489099f6169dd85b2683"
format_stamp: "Formatted at 2026-06-20 19:10:30 on dist-test-slave-n3fx"
server_key: "d298a744a28de114b2aab7fe73de1800"
server_key_iv: "0b0da11eca08a6f5b9f08a82279d97c9"
server_key_version: "encryptionkey@0"
I20260620 19:10:30.692065 17705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:30.708931 17705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:30.711987 17705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:30.712157 17705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:30.713685 17705 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:10:30.713740 17705 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:30.713779 17705 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:10:30.713794 17705 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 19:10:30 dist-test-slave-n3fx krb5kdc[17308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982630, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.155.67@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM
I20260620 19:10:30.723928 17705 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.67:42389
I20260620 19:10:30.724053 17835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.67:42389 every 8 connection(s)
I20260620 19:10:30.724979 17705 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
I20260620 19:10:30.726977 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 17705
I20260620 19:10:30.727123 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/minicluster-data/ts-2/wal/instance
I20260620 19:10:30.727372 17402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:30.714652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.67:55543 (local address 127.9.155.126:39669)
0620 19:10:30.714773 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:30.714776 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:30.715878 (+ 1102us) server_negotiation.cc:407] Connection header received
0620 19:10:30.716971 (+ 1093us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:30.716975 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:30.717039 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:30.717100 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:30.718459 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:30.719283 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:30.720656 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:30.720990 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:30.721110 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:30.723998 (+ 2888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:30.724002 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:30.724005 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:30.724035 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:30.725422 (+ 1387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:30.725992 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:30.725995 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:30.725996 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:30.726101 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:30.726369 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:30.726371 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:30.726372 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:30.726540 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:30.726608 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:30.727128 (+ 520us) server_negotiation.cc:299] Negotiation successful
0620 19:10:30.727252 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 19:10:30.727434 9837 external_mini_cluster.cc:1428] Setting key f8b28d6e88a7cb3e98809dd459f4322a
I20260620 19:10:30.728394 17729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:30.715203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:39669 (local address 127.9.155.67:55543)
0620 19:10:30.715718 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:30.715764 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:30.716722 (+ 958us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:30.717446 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:30.717453 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:30.717804 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:30.718320 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:30.718335 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:30.719421 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:30.719424 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:30.720551 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:30.720561 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:30.721150 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:30.721154 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:30.721324 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:30.721970 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:30.721989 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:30.723807 (+ 1818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:30.725555 (+ 1748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:30.725561 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:30.725575 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:30.725888 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:30.726170 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:30.726172 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:30.726173 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:30.726305 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:30.726613 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:30.726617 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:30.726879 (+ 262us) client_negotiation.cc:769] Sending connection context
0620 19:10:30.727077 (+ 198us) client_negotiation.cc:240] Negotiation successful
0620 19:10:30.727397 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":122,"threads_started":1}
I20260620 19:10:30.729451 17836 heartbeater.cc:347] Connected to a master server at 127.9.155.126:39669
I20260620 19:10:30.729590 17836 heartbeater.cc:464] Registering TS with master...
I20260620 19:10:30.730070 17836 heartbeater.cc:511] Master 127.9.155.126:39669 requested a full tablet report, sending...
I20260620 19:10:30.730993 17353 ts_manager.cc:194] Registered new tserver with Master: 3c96b775eb0f489099f6169dd85b2683 (127.9.155.67:42389)
I20260620 19:10:30.731668 17353 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.155.67@KRBTEST.COM'} at 127.9.155.67:55543
I20260620 19:10:30.737453 17402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:30.731367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.67:56109 (local address 127.9.155.126:39669)
0620 19:10:30.731476 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:30.731481 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:30.731639 (+ 158us) server_negotiation.cc:407] Connection header received
0620 19:10:30.731725 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:30.731728 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:30.731767 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:30.731838 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:30.732296 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:30.732829 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:30.733617 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:30.733827 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:30.733909 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:30.734901 (+ 992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:30.734904 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:30.734907 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:30.734931 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:30.735994 (+ 1063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:30.736400 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:30.736403 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:30.736405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:30.736477 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:30.736713 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:30.736716 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:30.736717 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:30.736878 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:30.736958 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:30.737201 (+ 243us) server_negotiation.cc:299] Negotiation successful
0620 19:10:30.737304 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 19:10:30.737463 17842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:30.731294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:39669 (local address 127.9.155.67:56109)
0620 19:10:30.731574 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:30.731585 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:30.731662 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:30.731860 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:30.731863 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:30.732082 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:30.732178 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:30.732183 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:30.732945 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:30.732948 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:30.733514 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:30.733521 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:30.733920 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:30.733924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:30.734018 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:30.734451 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:30.734462 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:30.734783 (+ 321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:30.736087 (+ 1304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:30.736091 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:30.736093 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:30.736322 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:30.736559 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:30.736562 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:30.736564 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:30.736639 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:30.736959 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:30.736961 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:30.737056 (+ 95us) client_negotiation.cc:769] Sending connection context
0620 19:10:30.737141 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 19:10:30.737280 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":86,"threads_started":1}
I20260620 19:10:30.741171 9837 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 19:10:30 dist-test-slave-n3fx krb5kdc[17308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982630, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 19:10:30 dist-test-slave-n3fx krb5kdc[17308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:10:30 dist-test-slave-n3fx krb5kdc[17308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
W20260620 19:10:30.763674 17402 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:10:30.757767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52094 (local address 127.9.155.126:39669)
0620 19:10:30.757879 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:30.757882 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:30.757894 (+ 12us) server_negotiation.cc:407] Connection header received
0620 19:10:30.757936 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:30.757938 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:30.757977 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:30.758088 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:30.758822 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:30.759346 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:30.760259 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:30.760464 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:30.760539 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:30.763554 (+ 3015us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52094: BlockingRecv error: recv got EOF from 127.0.0.1:52094 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Jun 20 19:10:30 dist-test-slave-n3fx krb5kdc[17308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982630, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM
I20260620 19:10:30.773229 17402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:30.765177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52100 (local address 127.9.155.126:39669)
0620 19:10:30.765293 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:30.765297 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:30.765483 (+ 186us) server_negotiation.cc:407] Connection header received
0620 19:10:30.765561 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:30.765563 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:30.765602 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:30.765684 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:30.766243 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:30.766832 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:30.767616 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:30.767838 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:30.767927 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:30.770245 (+ 2318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:30.770249 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:30.770251 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:30.770279 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:30.771638 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:30.772055 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:30.772058 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:30.772060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:30.772156 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:30.772460 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:30.772462 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:30.772463 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:30.772649 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:30.772761 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:30.772998 (+ 237us) server_negotiation.cc:299] Negotiation successful
0620 19:10:30.773105 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 19:10:30.776142 17353 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52100:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:10:30.779196 17353 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:10:30.793224 17858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:30.787605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.67:42389 (local address 127.0.0.1:57906)
0620 19:10:30.788362 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:30.788415 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:30.788557 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:30.789233 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:30.789239 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:30.789273 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:10:30.789432 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:30.789440 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:30.790521 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:30.790523 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:30.791769 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:30.791779 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:30.792677 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:30.792680 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:30.792779 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:30.792814 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 19:10:30.792915 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 19:10:30.793045 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"mutex_wait_us":3,"spinlock_wait_cycles":5504,"thread_start_us":60,"threads_started":1}
I20260620 19:10:30.793206 17860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:30.787785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57906 (local address 127.9.155.67:42389)
0620 19:10:30.788699 (+ 914us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:30.788712 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:30.788733 (+ 21us) server_negotiation.cc:407] Connection header received
0620 19:10:30.788854 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:30.788860 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:30.789079 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:30.789578 (+ 499us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:10:30.789703 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:30.790412 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:30.791896 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:30.792573 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:30.792666 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:30.792832 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:30.792961 (+ 129us) server_negotiation.cc:299] Negotiation successful
0620 19:10:30.793052 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":812,"thread_start_us":98,"threads_started":1}
I20260620 19:10:30.793277 17859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:30.787985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.65:45895 (local address 127.0.0.1:48450)
0620 19:10:30.788759 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:30.788771 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:30.788841 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:30.789522 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:30.789525 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:30.789542 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:10:30.789620 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:30.789625 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:30.790917 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:30.790919 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:30.791877 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:30.791887 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:30.792893 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:30.792896 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:30.792977 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:30.793005 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 19:10:30.793050 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 19:10:30.793118 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":718,"mutex_wait_us":4,"thread_start_us":81,"threads_started":1}
I20260620 19:10:30.793390 17861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:30.787739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.66:39779 (local address 127.0.0.1:48394)
0620 19:10:30.788396 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:30.788436 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:30.788549 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:30.789233 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:30.789239 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:30.789271 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:10:30.789454 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:30.789461 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:30.790887 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:30.790890 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:30.792039 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:30.792047 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:30.793050 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:30.793053 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:30.793129 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:30.793147 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 19:10:30.793188 (+ 41us) client_negotiation.cc:240] Negotiation successful
0620 19:10:30.793242 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"mutex_wait_us":82,"spinlock_wait_cycles":6144,"thread_start_us":136,"threads_started":1}
I20260620 19:10:30.793869 17863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:30.788329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48450 (local address 127.9.155.65:45895)
0620 19:10:30.789072 (+ 743us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:30.789078 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:30.789099 (+ 21us) server_negotiation.cc:407] Connection header received
0620 19:10:30.789170 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:30.789174 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:30.789347 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:30.789523 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:10:30.789717 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:30.790780 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:30.791978 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:30.792755 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:30.793464 (+ 709us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:30.793611 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:30.793659 (+ 48us) server_negotiation.cc:299] Negotiation successful
0620 19:10:30.793744 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":645,"thread_start_us":79,"threads_started":1}
I20260620 19:10:30.794023 17862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:30.788280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48394 (local address 127.9.155.66:39779)
0620 19:10:30.788794 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:30.788805 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:30.788823 (+ 18us) server_negotiation.cc:407] Connection header received
0620 19:10:30.788881 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:30.788885 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:30.789065 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:30.789347 (+ 282us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:10:30.789581 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:30.790770 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:30.792151 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:30.792924 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:30.793622 (+ 698us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:30.793763 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:30.793806 (+ 43us) server_negotiation.cc:299] Negotiation successful
0620 19:10:30.793884 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":101,"threads_started":1}
I20260620 19:10:30.796626 17630 tablet_service.cc:1467] Processing CreateTablet for tablet 33368e622d17457982a9d0cd18558756 (DEFAULT_TABLE table=test-table [id=3ba0962990f24448b5f2d19cc5ed7ef5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:10:30.796526 17770 tablet_service.cc:1467] Processing CreateTablet for tablet 33368e622d17457982a9d0cd18558756 (DEFAULT_TABLE table=test-table [id=3ba0962990f24448b5f2d19cc5ed7ef5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:10:30.796522 17490 tablet_service.cc:1467] Processing CreateTablet for tablet 33368e622d17457982a9d0cd18558756 (DEFAULT_TABLE table=test-table [id=3ba0962990f24448b5f2d19cc5ed7ef5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:10:30.797541 17490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33368e622d17457982a9d0cd18558756. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:30.797559 17770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33368e622d17457982a9d0cd18558756. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:30.797550 17630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33368e622d17457982a9d0cd18558756. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:30.804301 17864 tablet_bootstrap.cc:492] T 33368e622d17457982a9d0cd18558756 P 35a74422fbe4496abc46cf71d22af858: Bootstrap starting.
I20260620 19:10:30.805675 17865 tablet_bootstrap.cc:492] T 33368e622d17457982a9d0cd18558756 P 3c96b775eb0f489099f6169dd85b2683: Bootstrap starting.
I20260620 19:10:30.806592 17864 tablet_bootstrap.cc:654] T 33368e622d17457982a9d0cd18558756 P 35a74422fbe4496abc46cf71d22af858: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:30.807317 17864 log.cc:824] T 33368e622d17457982a9d0cd18558756 P 35a74422fbe4496abc46cf71d22af858: Log is configured to *not* fsync() on all Append() calls
I20260620 19:10:30.807480 17865 tablet_bootstrap.cc:654] T 33368e622d17457982a9d0cd18558756 P 3c96b775eb0f489099f6169dd85b2683: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:30.807905 17866 tablet_bootstrap.cc:492] T 33368e622d17457982a9d0cd18558756 P b22f4448136b4b06945ad3ffa8b0231a: Bootstrap starting.
I20260620 19:10:30.808317 17865 log.cc:824] T 33368e622d17457982a9d0cd18558756 P 3c96b775eb0f489099f6169dd85b2683: Log is configured to *not* fsync() on all Append() calls
I20260620 19:10:30.809496 17864 tablet_bootstrap.cc:492] T 33368e622d17457982a9d0cd18558756 P 35a74422fbe4496abc46cf71d22af858: No bootstrap required, opened a new log
I20260620 19:10:30.809689 17864 ts_tablet_manager.cc:1397] T 33368e622d17457982a9d0cd18558756 P 35a74422fbe4496abc46cf71d22af858: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 19:10:30.810061 17866 tablet_bootstrap.cc:654] T 33368e622d17457982a9d0cd18558756 P b22f4448136b4b06945ad3ffa8b0231a: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:30.810235 17865 tablet_bootstrap.cc:492] T 33368e622d17457982a9d0cd18558756 P 3c96b775eb0f489099f6169dd85b2683: No bootstrap required, opened a new log
I20260620 19:10:30.810420 17865 ts_tablet_manager.cc:1397] T 33368e622d17457982a9d0cd18558756 P 3c96b775eb0f489099f6169dd85b2683: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 19:10:30.810869 17866 log.cc:824] T 33368e622d17457982a9d0cd18558756 P b22f4448136b4b06945ad3ffa8b0231a: Log is configured to *not* fsync() on all Append() calls
I20260620 19:10:30.812932 17866 tablet_bootstrap.cc:492] T 33368e622d17457982a9d0cd18558756 P b22f4448136b4b06945ad3ffa8b0231a: No bootstrap required, opened a new log
I20260620 19:10:30.813146 17866 ts_tablet_manager.cc:1397] T 33368e622d17457982a9d0cd18558756 P b22f4448136b4b06945ad3ffa8b0231a: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260620 19:10:30.813611 17865 raft_consensus.cc:348] T 33368e622d17457982a9d0cd18558756 P 3c96b775eb0f489099f6169dd85b2683 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c96b775eb0f489099f6169dd85b2683" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 42389 } } peers { permanent_uuid: "b22f4448136b4b06945ad3ffa8b0231a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 39779 } } peers { permanent_uuid: "35a74422fbe4496abc46cf71d22af858" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 45895 } }
I20260620 19:10:30.813834 17865 raft_consensus.cc:374] T 33368e622d17457982a9d0cd18558756 P 3c96b775eb0f489099f6169dd85b2683 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:30.813885 17865 raft_consensus.cc:729] T 33368e622d17457982a9d0cd18558756 P 3c96b775eb0f489099f6169dd85b2683 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c96b775eb0f489099f6169dd85b2683, State: Initialized, Role: FOLLOWER
I20260620 19:10:30.813783 17864 raft_consensus.cc:348] T 33368e622d17457982a9d0cd18558756 P 35a74422fbe4496abc46cf71d22af858 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c96b775eb0f489099f6169dd85b2683" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 42389 } } peers { permanent_uuid: "b22f4448136b4b06945ad3ffa8b0231a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 39779 } } peers { permanent_uuid: "35a74422fbe4496abc46cf71d22af858" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 45895 } }
I20260620 19:10:30.814039 17864 raft_consensus.cc:374] T 33368e622d17457982a9d0cd18558756 P 35a74422fbe4496abc46cf71d22af858 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:30.814098 17864 raft_consensus.cc:729] T 33368e622d17457982a9d0cd18558756 P 35a74422fbe4496abc46cf71d22af858 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35a74422fbe4496abc46cf71d22af858, State: Initialized, Role: FOLLOWER
I20260620 19:10:30.814355 17865 consensus_queue.cc:260] T 33368e622d17457982a9d0cd18558756 P 3c96b775eb0f489099f6169dd85b2683 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c96b775eb0f489099f6169dd85b2683" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 42389 } } peers { permanent_uuid: "b22f4448136b4b06945ad3ffa8b0231a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 39779 } } peers { permanent_uuid: "35a74422fbe4496abc46cf71d22af858" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 45895 } }
I20260620 19:10:30.814683 17864 consensus_queue.cc:260] T 33368e622d17457982a9d0cd18558756 P 35a74422fbe4496abc46cf71d22af858 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c96b775eb0f489099f6169dd85b2683" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 42389 } } peers { permanent_uuid: "b22f4448136b4b06945ad3ffa8b0231a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 39779 } } peers { permanent_uuid: "35a74422fbe4496abc46cf71d22af858" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 45895 } }
I20260620 19:10:30.815089 17836 heartbeater.cc:503] Master 127.9.155.126:39669 was elected leader, sending a full tablet report...
I20260620 19:10:30.815327 17865 ts_tablet_manager.cc:1428] T 33368e622d17457982a9d0cd18558756 P 3c96b775eb0f489099f6169dd85b2683: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 19:10:30.815582 17864 ts_tablet_manager.cc:1428] T 33368e622d17457982a9d0cd18558756 P 35a74422fbe4496abc46cf71d22af858: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 19:10:30.815914 17556 heartbeater.cc:503] Master 127.9.155.126:39669 was elected leader, sending a full tablet report...
I20260620 19:10:30.818122 17866 raft_consensus.cc:348] T 33368e622d17457982a9d0cd18558756 P b22f4448136b4b06945ad3ffa8b0231a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c96b775eb0f489099f6169dd85b2683" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 42389 } } peers { permanent_uuid: "b22f4448136b4b06945ad3ffa8b0231a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 39779 } } peers { permanent_uuid: "35a74422fbe4496abc46cf71d22af858" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 45895 } }
I20260620 19:10:30.818400 17866 raft_consensus.cc:374] T 33368e622d17457982a9d0cd18558756 P b22f4448136b4b06945ad3ffa8b0231a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:30.818460 17866 raft_consensus.cc:729] T 33368e622d17457982a9d0cd18558756 P b22f4448136b4b06945ad3ffa8b0231a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22f4448136b4b06945ad3ffa8b0231a, State: Initialized, Role: FOLLOWER
I20260620 19:10:30.818851 17866 consensus_queue.cc:260] T 33368e622d17457982a9d0cd18558756 P b22f4448136b4b06945ad3ffa8b0231a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c96b775eb0f489099f6169dd85b2683" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 42389 } } peers { permanent_uuid: "b22f4448136b4b06945ad3ffa8b0231a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 39779 } } peers { permanent_uuid: "35a74422fbe4496abc46cf71d22af858" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 45895 } }
I20260620 19:10:30.819459 17696 heartbeater.cc:503] Master 127.9.155.126:39669 was elected leader, sending a full tablet report...
I20260620 19:10:30.819839 17866 ts_tablet_manager.cc:1428] T 33368e622d17457982a9d0cd18558756 P b22f4448136b4b06945ad3ffa8b0231a: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20260620 19:10:30.892453 17872 raft_consensus.cc:482] T 33368e622d17457982a9d0cd18558756 P b22f4448136b4b06945ad3ffa8b0231a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:10:30.892653 17872 raft_consensus.cc:504] T 33368e622d17457982a9d0cd18558756 P b22f4448136b4b06945ad3ffa8b0231a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c96b775eb0f489099f6169dd85b2683" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 42389 } } peers { permanent_uuid: "b22f4448136b4b06945ad3ffa8b0231a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 39779 } } peers { permanent_uuid: "35a74422fbe4496abc46cf71d22af858" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 45895 } }
I20260620 19:10:30.893833 17872 leader_election.cc:288] T 33368e622d17457982a9d0cd18558756 P b22f4448136b4b06945ad3ffa8b0231a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c96b775eb0f489099f6169dd85b2683 (127.9.155.67:42389), 35a74422fbe4496abc46cf71d22af858 (127.9.155.65:45895)
I20260620 19:10:30.897910 17589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:30.894142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.65:45895 (local address 127.9.155.66:52105)
0620 19:10:30.894344 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:30.894358 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:30.894469 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:30.894728 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:30.894731 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:30.894746 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:10:30.894863 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:30.894869 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:30.895694 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:30.895697 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:30.896776 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:30.896787 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:30.897566 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:30.897568 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:30.897649 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:30.897668 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 19:10:30.897723 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 19:10:30.897783 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":33}
I20260620 19:10:30.897941 17863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:30.894222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.66:52105 (local address 127.9.155.65:45895)
0620 19:10:30.894350 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:30.894355 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:30.894426 (+ 71us) server_negotiation.cc:407] Connection header received
0620 19:10:30.894578 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:30.894580 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:30.894623 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:30.894699 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:10:30.894987 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:30.895590 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:30.896925 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:30.897451 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:30.897573 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:30.897627 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:30.897738 (+ 111us) server_negotiation.cc:299] Negotiation successful
0620 19:10:30.897806 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 19:10:30.898586 17873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:30.894143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.67:42389 (local address 127.9.155.66:55429)
0620 19:10:30.894607 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:30.894618 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:30.894698 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:30.894922 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:30.894925 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:30.894942 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:10:30.895456 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:30.895466 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:30.896292 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:30.896296 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:30.897286 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:30.897295 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:30.898227 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:30.898231 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:30.898334 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:30.898356 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 19:10:30.898418 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 19:10:30.898472 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"mutex_wait_us":66,"thread_start_us":184,"threads_started":1}
I20260620 19:10:30.898650 17860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:30.894219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.66:55429 (local address 127.9.155.67:42389)
0620 19:10:30.894354 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:30.894359 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:30.894693 (+ 334us) server_negotiation.cc:407] Connection header received
0620 19:10:30.894779 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:30.894782 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:30.894829 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:30.894886 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:10:30.895566 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:30.896164 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:30.897438 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:30.897992 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:30.898101 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:30.898147 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:30.898467 (+ 320us) server_negotiation.cc:299] Negotiation successful
0620 19:10:30.898545 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 19:10:30.898682 17510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33368e622d17457982a9d0cd18558756" candidate_uuid: "b22f4448136b4b06945ad3ffa8b0231a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35a74422fbe4496abc46cf71d22af858" is_pre_election: true
I20260620 19:10:30.899040 17510 raft_consensus.cc:2454] T 33368e622d17457982a9d0cd18558756 P 35a74422fbe4496abc46cf71d22af858 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22f4448136b4b06945ad3ffa8b0231a in term 0.
I20260620 19:10:30.899186 17790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33368e622d17457982a9d0cd18558756" candidate_uuid: "b22f4448136b4b06945ad3ffa8b0231a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c96b775eb0f489099f6169dd85b2683" is_pre_election: true
I20260620 19:10:30.899470 17790 raft_consensus.cc:2454] T 33368e622d17457982a9d0cd18558756 P 3c96b775eb0f489099f6169dd85b2683 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22f4448136b4b06945ad3ffa8b0231a in term 0.
I20260620 19:10:30.899523 17581 leader_election.cc:302] T 33368e622d17457982a9d0cd18558756 P b22f4448136b4b06945ad3ffa8b0231a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35a74422fbe4496abc46cf71d22af858, b22f4448136b4b06945ad3ffa8b0231a; no voters:
I20260620 19:10:30.899791 17872 raft_consensus.cc:2781] T 33368e622d17457982a9d0cd18558756 P b22f4448136b4b06945ad3ffa8b0231a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:10:30.899868 17872 raft_consensus.cc:482] T 33368e622d17457982a9d0cd18558756 P b22f4448136b4b06945ad3ffa8b0231a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:10:30.899899 17872 raft_consensus.cc:3037] T 33368e622d17457982a9d0cd18558756 P b22f4448136b4b06945ad3ffa8b0231a [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:30.900949 17872 raft_consensus.cc:504] T 33368e622d17457982a9d0cd18558756 P b22f4448136b4b06945ad3ffa8b0231a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c96b775eb0f489099f6169dd85b2683" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 42389 } } peers { permanent_uuid: "b22f4448136b4b06945ad3ffa8b0231a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 39779 } } peers { permanent_uuid: "35a74422fbe4496abc46cf71d22af858" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 45895 } }
I20260620 19:10:30.901268 17872 leader_election.cc:288] T 33368e622d17457982a9d0cd18558756 P b22f4448136b4b06945ad3ffa8b0231a [CANDIDATE]: Term 1 election: Requested vote from peers 3c96b775eb0f489099f6169dd85b2683 (127.9.155.67:42389), 35a74422fbe4496abc46cf71d22af858 (127.9.155.65:45895)
I20260620 19:10:30.901609 17790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33368e622d17457982a9d0cd18558756" candidate_uuid: "b22f4448136b4b06945ad3ffa8b0231a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c96b775eb0f489099f6169dd85b2683"
I20260620 19:10:30.901623 17510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33368e622d17457982a9d0cd18558756" candidate_uuid: "b22f4448136b4b06945ad3ffa8b0231a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35a74422fbe4496abc46cf71d22af858"
I20260620 19:10:30.901716 17510 raft_consensus.cc:3037] T 33368e622d17457982a9d0cd18558756 P 35a74422fbe4496abc46cf71d22af858 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:30.901718 17790 raft_consensus.cc:3037] T 33368e622d17457982a9d0cd18558756 P 3c96b775eb0f489099f6169dd85b2683 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:30.902652 17510 raft_consensus.cc:2454] T 33368e622d17457982a9d0cd18558756 P 35a74422fbe4496abc46cf71d22af858 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22f4448136b4b06945ad3ffa8b0231a in term 1.
I20260620 19:10:30.902943 17581 leader_election.cc:302] T 33368e622d17457982a9d0cd18558756 P b22f4448136b4b06945ad3ffa8b0231a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35a74422fbe4496abc46cf71d22af858, b22f4448136b4b06945ad3ffa8b0231a; no voters:
I20260620 19:10:30.902999 17790 raft_consensus.cc:2454] T 33368e622d17457982a9d0cd18558756 P 3c96b775eb0f489099f6169dd85b2683 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22f4448136b4b06945ad3ffa8b0231a in term 1.
I20260620 19:10:30.903103 17872 raft_consensus.cc:2781] T 33368e622d17457982a9d0cd18558756 P b22f4448136b4b06945ad3ffa8b0231a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:10:30.903312 17872 raft_consensus.cc:686] T 33368e622d17457982a9d0cd18558756 P b22f4448136b4b06945ad3ffa8b0231a [term 1 LEADER]: Becoming Leader. State: Replica: b22f4448136b4b06945ad3ffa8b0231a, State: Running, Role: LEADER
I20260620 19:10:30.903635 17872 consensus_queue.cc:237] T 33368e622d17457982a9d0cd18558756 P b22f4448136b4b06945ad3ffa8b0231a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c96b775eb0f489099f6169dd85b2683" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 42389 } } peers { permanent_uuid: "b22f4448136b4b06945ad3ffa8b0231a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 39779 } } peers { permanent_uuid: "35a74422fbe4496abc46cf71d22af858" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 45895 } }
I20260620 19:10:30.906798 17353 catalog_manager.cc:5495] T 33368e622d17457982a9d0cd18558756 P b22f4448136b4b06945ad3ffa8b0231a reported cstate change: term changed from 0 to 1, leader changed from <none> to b22f4448136b4b06945ad3ffa8b0231a (127.9.155.66). New cstate: current_term: 1 leader_uuid: "b22f4448136b4b06945ad3ffa8b0231a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c96b775eb0f489099f6169dd85b2683" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 42389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b22f4448136b4b06945ad3ffa8b0231a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 39779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35a74422fbe4496abc46cf71d22af858" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 45895 } health_report { overall_health: UNKNOWN } } }
I20260620 19:10:30.945463 17862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:30.942041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48404 (local address 127.9.155.66:39779)
0620 19:10:30.942158 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:30.942161 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:30.942174 (+ 13us) server_negotiation.cc:407] Connection header received
0620 19:10:30.942270 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:30.942272 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:30.942324 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:30.942418 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:10:30.942647 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:30.943164 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:30.944036 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:30.944297 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:30.944396 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:30.944716 (+ 320us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:10:30.945171 (+ 455us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:10:30.945261 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:30.945295 (+ 34us) server_negotiation.cc:299] Negotiation successful
0620 19:10:30.945350 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 19:10:30.951609 17790 raft_consensus.cc:1261] T 33368e622d17457982a9d0cd18558756 P 3c96b775eb0f489099f6169dd85b2683 [term 1 FOLLOWER]: Refusing update from remote peer b22f4448136b4b06945ad3ffa8b0231a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:10:30.951609 17510 raft_consensus.cc:1261] T 33368e622d17457982a9d0cd18558756 P 35a74422fbe4496abc46cf71d22af858 [term 1 FOLLOWER]: Refusing update from remote peer b22f4448136b4b06945ad3ffa8b0231a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:10:30.952399 17872 consensus_queue.cc:1035] T 33368e622d17457982a9d0cd18558756 P b22f4448136b4b06945ad3ffa8b0231a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c96b775eb0f489099f6169dd85b2683" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 42389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:10:30.952608 17874 consensus_queue.cc:1035] T 33368e622d17457982a9d0cd18558756 P b22f4448136b4b06945ad3ffa8b0231a [LEADER]: Connected to new peer: Peer: permanent_uuid: "35a74422fbe4496abc46cf71d22af858" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 45895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:10:30.961627 17879 mvcc.cc:205] Tried to move back new op lower bound from 7299000856369414144 to 7299000856183836672. Current Snapshot: MvccSnapshot[applied={T|T < 7299000856369414144}]
I20260620 19:10:30.962512 17878 mvcc.cc:205] Tried to move back new op lower bound from 7299000856369414144 to 7299000856183836672. Current Snapshot: MvccSnapshot[applied={T|T < 7299000856369414144}]
I20260620 19:10:30.969266 17353 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52100:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:10:30.969467 17353 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52100:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:10:30.971951 17353 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 42920bad573b462593296d6a1276cb32: Sending DeleteTablet for 3 replicas of tablet 33368e622d17457982a9d0cd18558756
I20260620 19:10:30.973011 17630 tablet_service.cc:1514] Processing DeleteTablet for tablet 33368e622d17457982a9d0cd18558756 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:10:30 UTC) from {username='oryx', principal='oryx/127.9.155.126@KRBTEST.COM'} at 127.0.0.1:48394
I20260620 19:10:30.973506 17887 tablet_replica.cc:330] T 33368e622d17457982a9d0cd18558756 P b22f4448136b4b06945ad3ffa8b0231a: stopping tablet replica
I20260620 19:10:30.973529 17490 tablet_service.cc:1514] Processing DeleteTablet for tablet 33368e622d17457982a9d0cd18558756 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:10:30 UTC) from {username='oryx', principal='oryx/127.9.155.126@KRBTEST.COM'} at 127.0.0.1:48450
I20260620 19:10:30.973618 17353 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52100:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 19:10:30.973847 17887 raft_consensus.cc:2229] T 33368e622d17457982a9d0cd18558756 P b22f4448136b4b06945ad3ffa8b0231a [term 1 LEADER]: Raft consensus shutting down.
W20260620 19:10:30.974067 17353 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:10:30.974026 17770 tablet_service.cc:1514] Processing DeleteTablet for tablet 33368e622d17457982a9d0cd18558756 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:10:30 UTC) from {username='oryx', principal='oryx/127.9.155.126@KRBTEST.COM'} at 127.0.0.1:57906
I20260620 19:10:30.974272 17887 raft_consensus.cc:2258] T 33368e622d17457982a9d0cd18558756 P b22f4448136b4b06945ad3ffa8b0231a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:10:30.974540 17888 tablet_replica.cc:330] T 33368e622d17457982a9d0cd18558756 P 35a74422fbe4496abc46cf71d22af858: stopping tablet replica
I20260620 19:10:30.974820 17888 raft_consensus.cc:2229] T 33368e622d17457982a9d0cd18558756 P 35a74422fbe4496abc46cf71d22af858 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:10:30.975106 17888 raft_consensus.cc:2258] T 33368e622d17457982a9d0cd18558756 P 35a74422fbe4496abc46cf71d22af858 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:10:30.975835 17889 tablet_replica.cc:330] T 33368e622d17457982a9d0cd18558756 P 3c96b775eb0f489099f6169dd85b2683: stopping tablet replica
I20260620 19:10:30.976023 17887 ts_tablet_manager.cc:1905] T 33368e622d17457982a9d0cd18558756 P b22f4448136b4b06945ad3ffa8b0231a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:10:30.978127 17888 ts_tablet_manager.cc:1905] T 33368e622d17457982a9d0cd18558756 P 35a74422fbe4496abc46cf71d22af858: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:10:30.979067 17887 ts_tablet_manager.cc:1918] T 33368e622d17457982a9d0cd18558756 P b22f4448136b4b06945ad3ffa8b0231a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:10:30.979171 17887 log.cc:1196] T 33368e622d17457982a9d0cd18558756 P b22f4448136b4b06945ad3ffa8b0231a: Deleting WAL directory at /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/minicluster-data/ts-1/wal/wals/33368e622d17457982a9d0cd18558756
I20260620 19:10:30.979250 17630 tablet_service.cc:1467] Processing CreateTablet for tablet aff233c162034234ab68b9963000f08f (DEFAULT_TABLE table=test-table [id=f585d9db812a4612801666236d2ad51d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:10:30.979470 17887 ts_tablet_manager.cc:1939] T 33368e622d17457982a9d0cd18558756 P b22f4448136b4b06945ad3ffa8b0231a: Deleting consensus metadata
I20260620 19:10:30.979514 17630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aff233c162034234ab68b9963000f08f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:30.981084 17888 ts_tablet_manager.cc:1918] T 33368e622d17457982a9d0cd18558756 P 35a74422fbe4496abc46cf71d22af858: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:10:30.981184 17888 log.cc:1196] T 33368e622d17457982a9d0cd18558756 P 35a74422fbe4496abc46cf71d22af858: Deleting WAL directory at /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/minicluster-data/ts-0/wal/wals/33368e622d17457982a9d0cd18558756
I20260620 19:10:30.981470 17888 ts_tablet_manager.cc:1939] T 33368e622d17457982a9d0cd18558756 P 35a74422fbe4496abc46cf71d22af858: Deleting consensus metadata
I20260620 19:10:30.981722 17866 tablet_bootstrap.cc:492] T aff233c162034234ab68b9963000f08f P b22f4448136b4b06945ad3ffa8b0231a: Bootstrap starting.
I20260620 19:10:30.982707 17866 tablet_bootstrap.cc:654] T aff233c162034234ab68b9963000f08f P b22f4448136b4b06945ad3ffa8b0231a: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:30.982744 17340 catalog_manager.cc:4841] TS b22f4448136b4b06945ad3ffa8b0231a (127.9.155.66:39779): tablet 33368e622d17457982a9d0cd18558756 (table test-table [id=3ba0962990f24448b5f2d19cc5ed7ef5]) successfully deleted
I20260620 19:10:30.983306 17770 tablet_service.cc:1467] Processing CreateTablet for tablet aff233c162034234ab68b9963000f08f (DEFAULT_TABLE table=test-table [id=f585d9db812a4612801666236d2ad51d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:10:30.983582 17770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aff233c162034234ab68b9963000f08f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:30.983949 17866 tablet_bootstrap.cc:492] T aff233c162034234ab68b9963000f08f P b22f4448136b4b06945ad3ffa8b0231a: No bootstrap required, opened a new log
I20260620 19:10:30.984045 17866 ts_tablet_manager.cc:1397] T aff233c162034234ab68b9963000f08f P b22f4448136b4b06945ad3ffa8b0231a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260620 19:10:30.984496 17866 raft_consensus.cc:348] T aff233c162034234ab68b9963000f08f P b22f4448136b4b06945ad3ffa8b0231a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35a74422fbe4496abc46cf71d22af858" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 45895 } } peers { permanent_uuid: "b22f4448136b4b06945ad3ffa8b0231a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 39779 } } peers { permanent_uuid: "3c96b775eb0f489099f6169dd85b2683" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 42389 } }
I20260620 19:10:30.984611 17866 raft_consensus.cc:374] T aff233c162034234ab68b9963000f08f P b22f4448136b4b06945ad3ffa8b0231a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:30.984649 17866 raft_consensus.cc:729] T aff233c162034234ab68b9963000f08f P b22f4448136b4b06945ad3ffa8b0231a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22f4448136b4b06945ad3ffa8b0231a, State: Initialized, Role: FOLLOWER
I20260620 19:10:30.984769 17866 consensus_queue.cc:260] T aff233c162034234ab68b9963000f08f P b22f4448136b4b06945ad3ffa8b0231a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35a74422fbe4496abc46cf71d22af858" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 45895 } } peers { permanent_uuid: "b22f4448136b4b06945ad3ffa8b0231a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 39779 } } peers { permanent_uuid: "3c96b775eb0f489099f6169dd85b2683" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 42389 } }
I20260620 19:10:30.985042 17866 ts_tablet_manager.cc:1428] T aff233c162034234ab68b9963000f08f P b22f4448136b4b06945ad3ffa8b0231a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:10:30.985930 17865 tablet_bootstrap.cc:492] T aff233c162034234ab68b9963000f08f P 3c96b775eb0f489099f6169dd85b2683: Bootstrap starting.
I20260620 19:10:30.986521 17337 catalog_manager.cc:4841] TS 35a74422fbe4496abc46cf71d22af858 (127.9.155.65:45895): tablet 33368e622d17457982a9d0cd18558756 (table test-table [id=3ba0962990f24448b5f2d19cc5ed7ef5]) successfully deleted
I20260620 19:10:30.986937 17865 tablet_bootstrap.cc:654] T aff233c162034234ab68b9963000f08f P 3c96b775eb0f489099f6169dd85b2683: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:30.987406 17889 raft_consensus.cc:2229] T 33368e622d17457982a9d0cd18558756 P 3c96b775eb0f489099f6169dd85b2683 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:10:30.987779 17889 raft_consensus.cc:2258] T 33368e622d17457982a9d0cd18558756 P 3c96b775eb0f489099f6169dd85b2683 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:10:30.987831 17490 tablet_service.cc:1467] Processing CreateTablet for tablet aff233c162034234ab68b9963000f08f (DEFAULT_TABLE table=test-table [id=f585d9db812a4612801666236d2ad51d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:10:30.988081 17490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aff233c162034234ab68b9963000f08f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:30.989235 17865 tablet_bootstrap.cc:492] T aff233c162034234ab68b9963000f08f P 3c96b775eb0f489099f6169dd85b2683: No bootstrap required, opened a new log
I20260620 19:10:30.989319 17865 ts_tablet_manager.cc:1397] T aff233c162034234ab68b9963000f08f P 3c96b775eb0f489099f6169dd85b2683: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 19:10:30.989753 17865 raft_consensus.cc:348] T aff233c162034234ab68b9963000f08f P 3c96b775eb0f489099f6169dd85b2683 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35a74422fbe4496abc46cf71d22af858" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 45895 } } peers { permanent_uuid: "b22f4448136b4b06945ad3ffa8b0231a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 39779 } } peers { permanent_uuid: "3c96b775eb0f489099f6169dd85b2683" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 42389 } }
I20260620 19:10:30.989856 17865 raft_consensus.cc:374] T aff233c162034234ab68b9963000f08f P 3c96b775eb0f489099f6169dd85b2683 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:30.989882 17865 raft_consensus.cc:729] T aff233c162034234ab68b9963000f08f P 3c96b775eb0f489099f6169dd85b2683 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c96b775eb0f489099f6169dd85b2683, State: Initialized, Role: FOLLOWER
I20260620 19:10:30.989913 17889 ts_tablet_manager.cc:1905] T 33368e622d17457982a9d0cd18558756 P 3c96b775eb0f489099f6169dd85b2683: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:10:30.990002 17865 consensus_queue.cc:260] T aff233c162034234ab68b9963000f08f P 3c96b775eb0f489099f6169dd85b2683 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35a74422fbe4496abc46cf71d22af858" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 45895 } } peers { permanent_uuid: "b22f4448136b4b06945ad3ffa8b0231a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 39779 } } peers { permanent_uuid: "3c96b775eb0f489099f6169dd85b2683" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 42389 } }
I20260620 19:10:30.990235 17865 ts_tablet_manager.cc:1428] T aff233c162034234ab68b9963000f08f P 3c96b775eb0f489099f6169dd85b2683: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:10:30.990386 17864 tablet_bootstrap.cc:492] T aff233c162034234ab68b9963000f08f P 35a74422fbe4496abc46cf71d22af858: Bootstrap starting.
I20260620 19:10:30.991302 17864 tablet_bootstrap.cc:654] T aff233c162034234ab68b9963000f08f P 35a74422fbe4496abc46cf71d22af858: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:30.992511 17889 ts_tablet_manager.cc:1918] T 33368e622d17457982a9d0cd18558756 P 3c96b775eb0f489099f6169dd85b2683: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:10:30.992614 17889 log.cc:1196] T 33368e622d17457982a9d0cd18558756 P 3c96b775eb0f489099f6169dd85b2683: Deleting WAL directory at /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/minicluster-data/ts-2/wal/wals/33368e622d17457982a9d0cd18558756
I20260620 19:10:30.992898 17889 ts_tablet_manager.cc:1939] T 33368e622d17457982a9d0cd18558756 P 3c96b775eb0f489099f6169dd85b2683: Deleting consensus metadata
I20260620 19:10:30.993173 17864 tablet_bootstrap.cc:492] T aff233c162034234ab68b9963000f08f P 35a74422fbe4496abc46cf71d22af858: No bootstrap required, opened a new log
I20260620 19:10:30.993251 17864 ts_tablet_manager.cc:1397] T aff233c162034234ab68b9963000f08f P 35a74422fbe4496abc46cf71d22af858: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 19:10:30.993853 17864 raft_consensus.cc:348] T aff233c162034234ab68b9963000f08f P 35a74422fbe4496abc46cf71d22af858 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35a74422fbe4496abc46cf71d22af858" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 45895 } } peers { permanent_uuid: "b22f4448136b4b06945ad3ffa8b0231a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 39779 } } peers { permanent_uuid: "3c96b775eb0f489099f6169dd85b2683" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 42389 } }
I20260620 19:10:30.993963 17864 raft_consensus.cc:374] T aff233c162034234ab68b9963000f08f P 35a74422fbe4496abc46cf71d22af858 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:30.993995 17864 raft_consensus.cc:729] T aff233c162034234ab68b9963000f08f P 35a74422fbe4496abc46cf71d22af858 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35a74422fbe4496abc46cf71d22af858, State: Initialized, Role: FOLLOWER
I20260620 19:10:30.994118 17339 catalog_manager.cc:4841] TS 3c96b775eb0f489099f6169dd85b2683 (127.9.155.67:42389): tablet 33368e622d17457982a9d0cd18558756 (table test-table [id=3ba0962990f24448b5f2d19cc5ed7ef5]) successfully deleted
I20260620 19:10:30.994112 17864 consensus_queue.cc:260] T aff233c162034234ab68b9963000f08f P 35a74422fbe4496abc46cf71d22af858 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35a74422fbe4496abc46cf71d22af858" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 45895 } } peers { permanent_uuid: "b22f4448136b4b06945ad3ffa8b0231a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 39779 } } peers { permanent_uuid: "3c96b775eb0f489099f6169dd85b2683" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 42389 } }
I20260620 19:10:30.994354 17864 ts_tablet_manager.cc:1428] T aff233c162034234ab68b9963000f08f P 35a74422fbe4496abc46cf71d22af858: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20260620 19:10:31.011216 17697 tablet.cc:2369] T aff233c162034234ab68b9963000f08f P b22f4448136b4b06945ad3ffa8b0231a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:10:31.034116 17882 raft_consensus.cc:482] T aff233c162034234ab68b9963000f08f P b22f4448136b4b06945ad3ffa8b0231a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:10:31.034272 17882 raft_consensus.cc:504] T aff233c162034234ab68b9963000f08f P b22f4448136b4b06945ad3ffa8b0231a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35a74422fbe4496abc46cf71d22af858" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 45895 } } peers { permanent_uuid: "b22f4448136b4b06945ad3ffa8b0231a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 39779 } } peers { permanent_uuid: "3c96b775eb0f489099f6169dd85b2683" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 42389 } }
W20260620 19:10:31.034699 17557 tablet.cc:2369] T aff233c162034234ab68b9963000f08f P 35a74422fbe4496abc46cf71d22af858: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:10:31.034742 17882 leader_election.cc:288] T aff233c162034234ab68b9963000f08f P b22f4448136b4b06945ad3ffa8b0231a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 35a74422fbe4496abc46cf71d22af858 (127.9.155.65:45895), 3c96b775eb0f489099f6169dd85b2683 (127.9.155.67:42389)
I20260620 19:10:31.035130 17510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aff233c162034234ab68b9963000f08f" candidate_uuid: "b22f4448136b4b06945ad3ffa8b0231a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35a74422fbe4496abc46cf71d22af858" is_pre_election: true
I20260620 19:10:31.035267 17510 raft_consensus.cc:2454] T aff233c162034234ab68b9963000f08f P 35a74422fbe4496abc46cf71d22af858 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22f4448136b4b06945ad3ffa8b0231a in term 0.
I20260620 19:10:31.035256 17790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aff233c162034234ab68b9963000f08f" candidate_uuid: "b22f4448136b4b06945ad3ffa8b0231a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c96b775eb0f489099f6169dd85b2683" is_pre_election: true
I20260620 19:10:31.035485 17790 raft_consensus.cc:2454] T aff233c162034234ab68b9963000f08f P 3c96b775eb0f489099f6169dd85b2683 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22f4448136b4b06945ad3ffa8b0231a in term 0.
I20260620 19:10:31.035562 17581 leader_election.cc:302] T aff233c162034234ab68b9963000f08f P b22f4448136b4b06945ad3ffa8b0231a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35a74422fbe4496abc46cf71d22af858, b22f4448136b4b06945ad3ffa8b0231a; no voters:
I20260620 19:10:31.035728 17882 raft_consensus.cc:2781] T aff233c162034234ab68b9963000f08f P b22f4448136b4b06945ad3ffa8b0231a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:10:31.035796 17882 raft_consensus.cc:482] T aff233c162034234ab68b9963000f08f P b22f4448136b4b06945ad3ffa8b0231a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:10:31.035832 17882 raft_consensus.cc:3037] T aff233c162034234ab68b9963000f08f P b22f4448136b4b06945ad3ffa8b0231a [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:31.037412 17882 raft_consensus.cc:504] T aff233c162034234ab68b9963000f08f P b22f4448136b4b06945ad3ffa8b0231a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35a74422fbe4496abc46cf71d22af858" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 45895 } } peers { permanent_uuid: "b22f4448136b4b06945ad3ffa8b0231a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 39779 } } peers { permanent_uuid: "3c96b775eb0f489099f6169dd85b2683" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 42389 } }
I20260620 19:10:31.037787 17882 leader_election.cc:288] T aff233c162034234ab68b9963000f08f P b22f4448136b4b06945ad3ffa8b0231a [CANDIDATE]: Term 1 election: Requested vote from peers 35a74422fbe4496abc46cf71d22af858 (127.9.155.65:45895), 3c96b775eb0f489099f6169dd85b2683 (127.9.155.67:42389)
I20260620 19:10:31.038151 17790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aff233c162034234ab68b9963000f08f" candidate_uuid: "b22f4448136b4b06945ad3ffa8b0231a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c96b775eb0f489099f6169dd85b2683"
I20260620 19:10:31.038190 17510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aff233c162034234ab68b9963000f08f" candidate_uuid: "b22f4448136b4b06945ad3ffa8b0231a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35a74422fbe4496abc46cf71d22af858"
I20260620 19:10:31.038287 17790 raft_consensus.cc:3037] T aff233c162034234ab68b9963000f08f P 3c96b775eb0f489099f6169dd85b2683 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:31.038293 17510 raft_consensus.cc:3037] T aff233c162034234ab68b9963000f08f P 35a74422fbe4496abc46cf71d22af858 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:31.039422 17510 raft_consensus.cc:2454] T aff233c162034234ab68b9963000f08f P 35a74422fbe4496abc46cf71d22af858 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22f4448136b4b06945ad3ffa8b0231a in term 1.
I20260620 19:10:31.039422 17790 raft_consensus.cc:2454] T aff233c162034234ab68b9963000f08f P 3c96b775eb0f489099f6169dd85b2683 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22f4448136b4b06945ad3ffa8b0231a in term 1.
I20260620 19:10:31.039731 17581 leader_election.cc:302] T aff233c162034234ab68b9963000f08f P b22f4448136b4b06945ad3ffa8b0231a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35a74422fbe4496abc46cf71d22af858, b22f4448136b4b06945ad3ffa8b0231a; no voters:
I20260620 19:10:31.039909 17882 raft_consensus.cc:2781] T aff233c162034234ab68b9963000f08f P b22f4448136b4b06945ad3ffa8b0231a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:10:31.039999 17882 raft_consensus.cc:686] T aff233c162034234ab68b9963000f08f P b22f4448136b4b06945ad3ffa8b0231a [term 1 LEADER]: Becoming Leader. State: Replica: b22f4448136b4b06945ad3ffa8b0231a, State: Running, Role: LEADER
I20260620 19:10:31.040131 17882 consensus_queue.cc:237] T aff233c162034234ab68b9963000f08f P b22f4448136b4b06945ad3ffa8b0231a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35a74422fbe4496abc46cf71d22af858" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 45895 } } peers { permanent_uuid: "b22f4448136b4b06945ad3ffa8b0231a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 39779 } } peers { permanent_uuid: "3c96b775eb0f489099f6169dd85b2683" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 42389 } }
I20260620 19:10:31.041896 17351 catalog_manager.cc:5495] T aff233c162034234ab68b9963000f08f P b22f4448136b4b06945ad3ffa8b0231a reported cstate change: term changed from 0 to 1, leader changed from <none> to b22f4448136b4b06945ad3ffa8b0231a (127.9.155.66). New cstate: current_term: 1 leader_uuid: "b22f4448136b4b06945ad3ffa8b0231a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35a74422fbe4496abc46cf71d22af858" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 45895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b22f4448136b4b06945ad3ffa8b0231a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 39779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c96b775eb0f489099f6169dd85b2683" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 42389 } health_report { overall_health: UNKNOWN } } }
Jun 20 19:10:31 dist-test-slave-n3fx krb5kdc[17308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982629, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.155.126@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM
I20260620 19:10:31.094760 17402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:31.086034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52116 (local address 127.9.155.126:39669)
0620 19:10:31.086153 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:31.086158 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:31.086355 (+ 197us) server_negotiation.cc:407] Connection header received
0620 19:10:31.086480 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:31.086483 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:31.086521 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:31.086587 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:31.087279 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:31.087850 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:31.088824 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:31.089039 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:31.089123 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:31.091698 (+ 2575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:31.091701 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:31.091702 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:31.091743 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:31.093134 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:31.093585 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:31.093589 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:31.093591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:31.093660 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:31.093893 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:31.093896 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:31.093898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:31.094080 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:31.094162 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:31.094444 (+ 282us) server_negotiation.cc:299] Negotiation successful
0620 19:10:31.094636 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":87,"server-negotiator.queue_time_us":43}
I20260620 19:10:31.094784 17899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:31.085983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:39669 (local address 127.0.0.1:52116)
0620 19:10:31.086261 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:31.086272 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:31.086396 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:31.086612 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:31.086615 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:31.087058 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:31.087178 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:31.087185 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:31.087944 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:31.087947 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:31.088728 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:31.088739 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:31.089145 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:31.089148 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:31.089222 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:31.089751 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:31.089763 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:31.091550 (+ 1787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:31.093239 (+ 1689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:31.093251 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:31.093259 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:31.093497 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:31.093734 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:31.093737 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:31.093739 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:31.093820 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:31.094173 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:31.094180 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:31.094287 (+ 107us) client_negotiation.cc:769] Sending connection context
0620 19:10:31.094381 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 19:10:31.094561 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":118,"threads_started":1}
I20260620 19:10:31.099486 17351 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.9.155.126@KRBTEST.COM'} at 127.0.0.1:52116:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 19:10:31.100421 17351 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:10:31.106046 17490 tablet_service.cc:1467] Processing CreateTablet for tablet c76508d7446b4fceb93a26732aecbc4d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=eaa078fc7b7b41f69d03f9edfdc9f343]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 19:10:31.106153 17770 tablet_service.cc:1467] Processing CreateTablet for tablet c76508d7446b4fceb93a26732aecbc4d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=eaa078fc7b7b41f69d03f9edfdc9f343]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 19:10:31.106153 17630 tablet_service.cc:1467] Processing CreateTablet for tablet c76508d7446b4fceb93a26732aecbc4d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=eaa078fc7b7b41f69d03f9edfdc9f343]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 19:10:31.106540 17630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c76508d7446b4fceb93a26732aecbc4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:31.106540 17770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c76508d7446b4fceb93a26732aecbc4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:31.106537 17490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c76508d7446b4fceb93a26732aecbc4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:31.109454 17866 tablet_bootstrap.cc:492] T c76508d7446b4fceb93a26732aecbc4d P b22f4448136b4b06945ad3ffa8b0231a: Bootstrap starting.
I20260620 19:10:31.109519 17865 tablet_bootstrap.cc:492] T c76508d7446b4fceb93a26732aecbc4d P 3c96b775eb0f489099f6169dd85b2683: Bootstrap starting.
I20260620 19:10:31.110492 17865 tablet_bootstrap.cc:654] T c76508d7446b4fceb93a26732aecbc4d P 3c96b775eb0f489099f6169dd85b2683: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:31.110786 17866 tablet_bootstrap.cc:654] T c76508d7446b4fceb93a26732aecbc4d P b22f4448136b4b06945ad3ffa8b0231a: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:31.111176 17864 tablet_bootstrap.cc:492] T c76508d7446b4fceb93a26732aecbc4d P 35a74422fbe4496abc46cf71d22af858: Bootstrap starting.
I20260620 19:10:31.111996 17866 tablet_bootstrap.cc:492] T c76508d7446b4fceb93a26732aecbc4d P b22f4448136b4b06945ad3ffa8b0231a: No bootstrap required, opened a new log
I20260620 19:10:31.112057 17864 tablet_bootstrap.cc:654] T c76508d7446b4fceb93a26732aecbc4d P 35a74422fbe4496abc46cf71d22af858: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:31.112071 17866 ts_tablet_manager.cc:1397] T c76508d7446b4fceb93a26732aecbc4d P b22f4448136b4b06945ad3ffa8b0231a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 19:10:31.112555 17866 raft_consensus.cc:348] T c76508d7446b4fceb93a26732aecbc4d P b22f4448136b4b06945ad3ffa8b0231a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35a74422fbe4496abc46cf71d22af858" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 45895 } } peers { permanent_uuid: "b22f4448136b4b06945ad3ffa8b0231a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 39779 } } peers { permanent_uuid: "3c96b775eb0f489099f6169dd85b2683" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 42389 } }
I20260620 19:10:31.112676 17866 raft_consensus.cc:374] T c76508d7446b4fceb93a26732aecbc4d P b22f4448136b4b06945ad3ffa8b0231a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:31.112703 17866 raft_consensus.cc:729] T c76508d7446b4fceb93a26732aecbc4d P b22f4448136b4b06945ad3ffa8b0231a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22f4448136b4b06945ad3ffa8b0231a, State: Initialized, Role: FOLLOWER
I20260620 19:10:31.112833 17866 consensus_queue.cc:260] T c76508d7446b4fceb93a26732aecbc4d P b22f4448136b4b06945ad3ffa8b0231a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35a74422fbe4496abc46cf71d22af858" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 45895 } } peers { permanent_uuid: "b22f4448136b4b06945ad3ffa8b0231a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 39779 } } peers { permanent_uuid: "3c96b775eb0f489099f6169dd85b2683" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 42389 } }
I20260620 19:10:31.113080 17882 tablet_replica.cc:439] T c76508d7446b4fceb93a26732aecbc4d P b22f4448136b4b06945ad3ffa8b0231a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35a74422fbe4496abc46cf71d22af858" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 45895 } } peers { permanent_uuid: "b22f4448136b4b06945ad3ffa8b0231a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 39779 } } peers { permanent_uuid: "3c96b775eb0f489099f6169dd85b2683" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 42389 } } }
I20260620 19:10:31.113232 17882 tablet_replica.cc:442] T c76508d7446b4fceb93a26732aecbc4d P b22f4448136b4b06945ad3ffa8b0231a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:10:31.113070 17866 ts_tablet_manager.cc:1428] T c76508d7446b4fceb93a26732aecbc4d P b22f4448136b4b06945ad3ffa8b0231a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:10:31.114470 17865 tablet_bootstrap.cc:492] T c76508d7446b4fceb93a26732aecbc4d P 3c96b775eb0f489099f6169dd85b2683: No bootstrap required, opened a new log
I20260620 19:10:31.114557 17865 ts_tablet_manager.cc:1397] T c76508d7446b4fceb93a26732aecbc4d P 3c96b775eb0f489099f6169dd85b2683: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260620 19:10:31.115044 17865 raft_consensus.cc:348] T c76508d7446b4fceb93a26732aecbc4d P 3c96b775eb0f489099f6169dd85b2683 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35a74422fbe4496abc46cf71d22af858" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 45895 } } peers { permanent_uuid: "b22f4448136b4b06945ad3ffa8b0231a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 39779 } } peers { permanent_uuid: "3c96b775eb0f489099f6169dd85b2683" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 42389 } }
I20260620 19:10:31.115168 17865 raft_consensus.cc:374] T c76508d7446b4fceb93a26732aecbc4d P 3c96b775eb0f489099f6169dd85b2683 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:31.115200 17865 raft_consensus.cc:729] T c76508d7446b4fceb93a26732aecbc4d P 3c96b775eb0f489099f6169dd85b2683 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c96b775eb0f489099f6169dd85b2683, State: Initialized, Role: FOLLOWER
I20260620 19:10:31.115322 17865 consensus_queue.cc:260] T c76508d7446b4fceb93a26732aecbc4d P 3c96b775eb0f489099f6169dd85b2683 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35a74422fbe4496abc46cf71d22af858" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 45895 } } peers { permanent_uuid: "b22f4448136b4b06945ad3ffa8b0231a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 39779 } } peers { permanent_uuid: "3c96b775eb0f489099f6169dd85b2683" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 42389 } }
I20260620 19:10:31.115584 17865 ts_tablet_manager.cc:1428] T c76508d7446b4fceb93a26732aecbc4d P 3c96b775eb0f489099f6169dd85b2683: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:10:31.115564 17870 tablet_replica.cc:439] T c76508d7446b4fceb93a26732aecbc4d P 3c96b775eb0f489099f6169dd85b2683: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35a74422fbe4496abc46cf71d22af858" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 45895 } } peers { permanent_uuid: "b22f4448136b4b06945ad3ffa8b0231a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 39779 } } peers { permanent_uuid: "3c96b775eb0f489099f6169dd85b2683" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 42389 } } }
I20260620 19:10:31.115713 17870 tablet_replica.cc:442] T c76508d7446b4fceb93a26732aecbc4d P 3c96b775eb0f489099f6169dd85b2683: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:10:31.116581 17864 tablet_bootstrap.cc:492] T c76508d7446b4fceb93a26732aecbc4d P 35a74422fbe4496abc46cf71d22af858: No bootstrap required, opened a new log
I20260620 19:10:31.116680 17864 ts_tablet_manager.cc:1397] T c76508d7446b4fceb93a26732aecbc4d P 35a74422fbe4496abc46cf71d22af858: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260620 19:10:31.117149 17864 raft_consensus.cc:348] T c76508d7446b4fceb93a26732aecbc4d P 35a74422fbe4496abc46cf71d22af858 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35a74422fbe4496abc46cf71d22af858" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 45895 } } peers { permanent_uuid: "b22f4448136b4b06945ad3ffa8b0231a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 39779 } } peers { permanent_uuid: "3c96b775eb0f489099f6169dd85b2683" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 42389 } }
I20260620 19:10:31.117259 17864 raft_consensus.cc:374] T c76508d7446b4fceb93a26732aecbc4d P 35a74422fbe4496abc46cf71d22af858 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:31.117290 17864 raft_consensus.cc:729] T c76508d7446b4fceb93a26732aecbc4d P 35a74422fbe4496abc46cf71d22af858 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35a74422fbe4496abc46cf71d22af858, State: Initialized, Role: FOLLOWER
I20260620 19:10:31.117409 17864 consensus_queue.cc:260] T c76508d7446b4fceb93a26732aecbc4d P 35a74422fbe4496abc46cf71d22af858 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35a74422fbe4496abc46cf71d22af858" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 45895 } } peers { permanent_uuid: "b22f4448136b4b06945ad3ffa8b0231a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 39779 } } peers { permanent_uuid: "3c96b775eb0f489099f6169dd85b2683" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 42389 } }
I20260620 19:10:31.117666 17864 ts_tablet_manager.cc:1428] T c76508d7446b4fceb93a26732aecbc4d P 35a74422fbe4496abc46cf71d22af858: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:10:31.117904 17871 tablet_replica.cc:439] T c76508d7446b4fceb93a26732aecbc4d P 35a74422fbe4496abc46cf71d22af858: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35a74422fbe4496abc46cf71d22af858" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 45895 } } peers { permanent_uuid: "b22f4448136b4b06945ad3ffa8b0231a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 39779 } } peers { permanent_uuid: "3c96b775eb0f489099f6169dd85b2683" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 42389 } } }
I20260620 19:10:31.118036 17871 tablet_replica.cc:442] T c76508d7446b4fceb93a26732aecbc4d P 35a74422fbe4496abc46cf71d22af858: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:10:31.200333 17870 raft_consensus.cc:482] T c76508d7446b4fceb93a26732aecbc4d P 3c96b775eb0f489099f6169dd85b2683 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:10:31.200495 17870 raft_consensus.cc:504] T c76508d7446b4fceb93a26732aecbc4d P 3c96b775eb0f489099f6169dd85b2683 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35a74422fbe4496abc46cf71d22af858" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 45895 } } peers { permanent_uuid: "b22f4448136b4b06945ad3ffa8b0231a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 39779 } } peers { permanent_uuid: "3c96b775eb0f489099f6169dd85b2683" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 42389 } }
I20260620 19:10:31.201548 17870 leader_election.cc:288] T c76508d7446b4fceb93a26732aecbc4d P 3c96b775eb0f489099f6169dd85b2683 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 35a74422fbe4496abc46cf71d22af858 (127.9.155.65:45895), b22f4448136b4b06945ad3ffa8b0231a (127.9.155.66:39779)
I20260620 19:10:31.205687 17729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:31.201781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.65:45895 (local address 127.9.155.67:58321)
0620 19:10:31.201902 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:31.201942 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:31.202041 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:31.202414 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:31.202417 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:31.202433 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:10:31.202556 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:31.202563 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:31.203348 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:31.203351 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:31.204514 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:31.204526 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:31.205319 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:31.205322 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:31.205397 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:31.205420 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 19:10:31.205483 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 19:10:31.205551 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 19:10:31.205734 17863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:31.201993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.67:58321 (local address 127.9.155.65:45895)
0620 19:10:31.202119 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:31.202124 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:31.202140 (+ 16us) server_negotiation.cc:407] Connection header received
0620 19:10:31.202241 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:31.202243 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:31.202293 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:31.202396 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:10:31.202670 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:31.203236 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:31.204653 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:31.205197 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:31.205319 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:31.205376 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:31.205512 (+ 136us) server_negotiation.cc:299] Negotiation successful
0620 19:10:31.205592 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 19:10:31.206067 17903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:31.201857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.66:39779 (local address 127.9.155.67:37089)
0620 19:10:31.202192 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:31.202202 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:31.202287 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:31.202511 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:31.202513 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:31.202528 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:10:31.202959 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:31.202967 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:31.203808 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:31.203812 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:31.204867 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:31.204874 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:31.205677 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:31.205680 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:31.205774 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:31.205796 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 19:10:31.205863 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 19:10:31.205937 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":68,"threads_started":1}
I20260620 19:10:31.206094 17862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:31.202009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.67:37089 (local address 127.9.155.66:39779)
0620 19:10:31.202121 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:31.202125 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:31.202277 (+ 152us) server_negotiation.cc:407] Connection header received
0620 19:10:31.202359 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:31.202363 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:31.202421 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:31.202489 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:10:31.203071 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:31.203693 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:31.204984 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:31.205555 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:31.205685 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:31.205734 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:31.205894 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 19:10:31.205989 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 19:10:31.206360 17510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c76508d7446b4fceb93a26732aecbc4d" candidate_uuid: "3c96b775eb0f489099f6169dd85b2683" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35a74422fbe4496abc46cf71d22af858" is_pre_election: true
I20260620 19:10:31.206528 17510 raft_consensus.cc:2454] T c76508d7446b4fceb93a26732aecbc4d P 35a74422fbe4496abc46cf71d22af858 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c96b775eb0f489099f6169dd85b2683 in term 0.
I20260620 19:10:31.206754 17650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c76508d7446b4fceb93a26732aecbc4d" candidate_uuid: "3c96b775eb0f489099f6169dd85b2683" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22f4448136b4b06945ad3ffa8b0231a" is_pre_election: true
I20260620 19:10:31.206935 17721 leader_election.cc:302] T c76508d7446b4fceb93a26732aecbc4d P 3c96b775eb0f489099f6169dd85b2683 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35a74422fbe4496abc46cf71d22af858, 3c96b775eb0f489099f6169dd85b2683; no voters:
I20260620 19:10:31.207029 17650 raft_consensus.cc:2454] T c76508d7446b4fceb93a26732aecbc4d P b22f4448136b4b06945ad3ffa8b0231a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c96b775eb0f489099f6169dd85b2683 in term 0.
I20260620 19:10:31.207237 17870 raft_consensus.cc:2781] T c76508d7446b4fceb93a26732aecbc4d P 3c96b775eb0f489099f6169dd85b2683 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:10:31.207314 17870 raft_consensus.cc:482] T c76508d7446b4fceb93a26732aecbc4d P 3c96b775eb0f489099f6169dd85b2683 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:10:31.207372 17870 raft_consensus.cc:3037] T c76508d7446b4fceb93a26732aecbc4d P 3c96b775eb0f489099f6169dd85b2683 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:31.208563 17870 raft_consensus.cc:504] T c76508d7446b4fceb93a26732aecbc4d P 3c96b775eb0f489099f6169dd85b2683 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35a74422fbe4496abc46cf71d22af858" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 45895 } } peers { permanent_uuid: "b22f4448136b4b06945ad3ffa8b0231a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 39779 } } peers { permanent_uuid: "3c96b775eb0f489099f6169dd85b2683" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 42389 } }
I20260620 19:10:31.208952 17870 leader_election.cc:288] T c76508d7446b4fceb93a26732aecbc4d P 3c96b775eb0f489099f6169dd85b2683 [CANDIDATE]: Term 1 election: Requested vote from peers 35a74422fbe4496abc46cf71d22af858 (127.9.155.65:45895), b22f4448136b4b06945ad3ffa8b0231a (127.9.155.66:39779)
I20260620 19:10:31.209647 17510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c76508d7446b4fceb93a26732aecbc4d" candidate_uuid: "3c96b775eb0f489099f6169dd85b2683" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35a74422fbe4496abc46cf71d22af858"
I20260620 19:10:31.209802 17510 raft_consensus.cc:3037] T c76508d7446b4fceb93a26732aecbc4d P 35a74422fbe4496abc46cf71d22af858 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:31.209966 17650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c76508d7446b4fceb93a26732aecbc4d" candidate_uuid: "3c96b775eb0f489099f6169dd85b2683" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22f4448136b4b06945ad3ffa8b0231a"
I20260620 19:10:31.210098 17650 raft_consensus.cc:3037] T c76508d7446b4fceb93a26732aecbc4d P b22f4448136b4b06945ad3ffa8b0231a [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:31.210815 17510 raft_consensus.cc:2454] T c76508d7446b4fceb93a26732aecbc4d P 35a74422fbe4496abc46cf71d22af858 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c96b775eb0f489099f6169dd85b2683 in term 1.
I20260620 19:10:31.211151 17650 raft_consensus.cc:2454] T c76508d7446b4fceb93a26732aecbc4d P b22f4448136b4b06945ad3ffa8b0231a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c96b775eb0f489099f6169dd85b2683 in term 1.
I20260620 19:10:31.211217 17721 leader_election.cc:302] T c76508d7446b4fceb93a26732aecbc4d P 3c96b775eb0f489099f6169dd85b2683 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35a74422fbe4496abc46cf71d22af858, 3c96b775eb0f489099f6169dd85b2683; no voters:
I20260620 19:10:31.211442 17870 raft_consensus.cc:2781] T c76508d7446b4fceb93a26732aecbc4d P 3c96b775eb0f489099f6169dd85b2683 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:10:31.211663 17870 raft_consensus.cc:686] T c76508d7446b4fceb93a26732aecbc4d P 3c96b775eb0f489099f6169dd85b2683 [term 1 LEADER]: Becoming Leader. State: Replica: 3c96b775eb0f489099f6169dd85b2683, State: Running, Role: LEADER
I20260620 19:10:31.211920 17870 consensus_queue.cc:237] T c76508d7446b4fceb93a26732aecbc4d P 3c96b775eb0f489099f6169dd85b2683 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35a74422fbe4496abc46cf71d22af858" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 45895 } } peers { permanent_uuid: "b22f4448136b4b06945ad3ffa8b0231a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 39779 } } peers { permanent_uuid: "3c96b775eb0f489099f6169dd85b2683" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 42389 } }
I20260620 19:10:31.213248 17904 tablet_replica.cc:439] T c76508d7446b4fceb93a26732aecbc4d P 3c96b775eb0f489099f6169dd85b2683: TxnStatusTablet state changed. Reason: New leader 3c96b775eb0f489099f6169dd85b2683. Latest consensus state: current_term: 1 leader_uuid: "3c96b775eb0f489099f6169dd85b2683" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35a74422fbe4496abc46cf71d22af858" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 45895 } } peers { permanent_uuid: "b22f4448136b4b06945ad3ffa8b0231a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 39779 } } peers { permanent_uuid: "3c96b775eb0f489099f6169dd85b2683" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 42389 } } }
I20260620 19:10:31.213392 17904 tablet_replica.cc:442] T c76508d7446b4fceb93a26732aecbc4d P 3c96b775eb0f489099f6169dd85b2683: This TxnStatusTablet replica's current role is: LEADER
I20260620 19:10:31.213721 17906 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 19:10:31.213888 17906 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 19:10:31.214689 17351 catalog_manager.cc:5495] T c76508d7446b4fceb93a26732aecbc4d P 3c96b775eb0f489099f6169dd85b2683 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c96b775eb0f489099f6169dd85b2683 (127.9.155.67). New cstate: current_term: 1 leader_uuid: "3c96b775eb0f489099f6169dd85b2683" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35a74422fbe4496abc46cf71d22af858" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 45895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b22f4448136b4b06945ad3ffa8b0231a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 39779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c96b775eb0f489099f6169dd85b2683" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 42389 } health_report { overall_health: HEALTHY } } }
W20260620 19:10:31.226411 17837 tablet.cc:2369] T aff233c162034234ab68b9963000f08f P 3c96b775eb0f489099f6169dd85b2683: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:10:31.244451 17899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:31.240770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.67:42389 (local address 127.0.0.1:57914)
0620 19:10:31.240982 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:31.240998 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:31.241110 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:31.241406 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:31.241409 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:31.241416 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 19:10:31.241522 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:31.241528 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:31.242223 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:31.242226 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:31.242844 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:31.242852 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:31.243281 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:31.243285 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:31.243375 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:31.243435 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 19:10:31.244097 (+ 662us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 19:10:31.244111 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 19:10:31.244175 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 19:10:31.244299 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115}
I20260620 19:10:31.244427 17860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:31.240877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57914 (local address 127.9.155.67:42389)
0620 19:10:31.241010 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:31.241014 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:31.241045 (+ 31us) server_negotiation.cc:407] Connection header received
0620 19:10:31.241243 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:31.241246 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:31.241299 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:31.241378 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:10:31.241617 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:31.242124 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:31.242953 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:31.243181 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:31.243268 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:31.243567 (+ 299us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:10:31.244007 (+ 440us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:10:31.244111 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:31.244185 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 19:10:31.244273 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 19:10:31.247684 17650 raft_consensus.cc:1261] T c76508d7446b4fceb93a26732aecbc4d P b22f4448136b4b06945ad3ffa8b0231a [term 1 FOLLOWER]: Refusing update from remote peer 3c96b775eb0f489099f6169dd85b2683: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:10:31.247718 17510 raft_consensus.cc:1261] T c76508d7446b4fceb93a26732aecbc4d P 35a74422fbe4496abc46cf71d22af858 [term 1 FOLLOWER]: Refusing update from remote peer 3c96b775eb0f489099f6169dd85b2683: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:10:31.248308 17904 consensus_queue.cc:1035] T c76508d7446b4fceb93a26732aecbc4d P 3c96b775eb0f489099f6169dd85b2683 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35a74422fbe4496abc46cf71d22af858" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 45895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:10:31.248492 17870 consensus_queue.cc:1035] T c76508d7446b4fceb93a26732aecbc4d P 3c96b775eb0f489099f6169dd85b2683 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22f4448136b4b06945ad3ffa8b0231a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 39779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:10:31.250022 17871 tablet_replica.cc:439] T c76508d7446b4fceb93a26732aecbc4d P 35a74422fbe4496abc46cf71d22af858: TxnStatusTablet state changed. Reason: New leader 3c96b775eb0f489099f6169dd85b2683. Latest consensus state: current_term: 1 leader_uuid: "3c96b775eb0f489099f6169dd85b2683" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35a74422fbe4496abc46cf71d22af858" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 45895 } } peers { permanent_uuid: "b22f4448136b4b06945ad3ffa8b0231a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 39779 } } peers { permanent_uuid: "3c96b775eb0f489099f6169dd85b2683" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 42389 } } }
I20260620 19:10:31.250161 17871 tablet_replica.cc:442] T c76508d7446b4fceb93a26732aecbc4d P 35a74422fbe4496abc46cf71d22af858: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:10:31.251062 17882 tablet_replica.cc:439] T c76508d7446b4fceb93a26732aecbc4d P b22f4448136b4b06945ad3ffa8b0231a: TxnStatusTablet state changed. Reason: New leader 3c96b775eb0f489099f6169dd85b2683. Latest consensus state: current_term: 1 leader_uuid: "3c96b775eb0f489099f6169dd85b2683" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35a74422fbe4496abc46cf71d22af858" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 45895 } } peers { permanent_uuid: "b22f4448136b4b06945ad3ffa8b0231a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 39779 } } peers { permanent_uuid: "3c96b775eb0f489099f6169dd85b2683" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 42389 } } }
I20260620 19:10:31.251192 17882 tablet_replica.cc:442] T c76508d7446b4fceb93a26732aecbc4d P b22f4448136b4b06945ad3ffa8b0231a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:10:31.252815 17870 tablet_replica.cc:439] T c76508d7446b4fceb93a26732aecbc4d P 3c96b775eb0f489099f6169dd85b2683: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3c96b775eb0f489099f6169dd85b2683" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35a74422fbe4496abc46cf71d22af858" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 45895 } } peers { permanent_uuid: "b22f4448136b4b06945ad3ffa8b0231a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 39779 } } peers { permanent_uuid: "3c96b775eb0f489099f6169dd85b2683" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 42389 } } }
I20260620 19:10:31.252931 17870 tablet_replica.cc:442] T c76508d7446b4fceb93a26732aecbc4d P 3c96b775eb0f489099f6169dd85b2683: This TxnStatusTablet replica's current role is: LEADER
I20260620 19:10:31.253356 17877 mvcc.cc:205] Tried to move back new op lower bound from 7299000857584779264 to 7299000857446903808. Current Snapshot: MvccSnapshot[applied={T|T < 7299000857584779264}]
I20260620 19:10:31.253525 17882 tablet_replica.cc:439] T c76508d7446b4fceb93a26732aecbc4d P b22f4448136b4b06945ad3ffa8b0231a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3c96b775eb0f489099f6169dd85b2683" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35a74422fbe4496abc46cf71d22af858" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 45895 } } peers { permanent_uuid: "b22f4448136b4b06945ad3ffa8b0231a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 39779 } } peers { permanent_uuid: "3c96b775eb0f489099f6169dd85b2683" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 42389 } } }
I20260620 19:10:31.253618 17882 tablet_replica.cc:442] T c76508d7446b4fceb93a26732aecbc4d P b22f4448136b4b06945ad3ffa8b0231a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:10:31.255388 17870 tablet_replica.cc:439] T c76508d7446b4fceb93a26732aecbc4d P 3c96b775eb0f489099f6169dd85b2683: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3c96b775eb0f489099f6169dd85b2683" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35a74422fbe4496abc46cf71d22af858" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 45895 } } peers { permanent_uuid: "b22f4448136b4b06945ad3ffa8b0231a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 39779 } } peers { permanent_uuid: "3c96b775eb0f489099f6169dd85b2683" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 42389 } } }
I20260620 19:10:31.255498 17870 tablet_replica.cc:442] T c76508d7446b4fceb93a26732aecbc4d P 3c96b775eb0f489099f6169dd85b2683: This TxnStatusTablet replica's current role is: LEADER
I20260620 19:10:31.256332 17871 tablet_replica.cc:439] T c76508d7446b4fceb93a26732aecbc4d P 35a74422fbe4496abc46cf71d22af858: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3c96b775eb0f489099f6169dd85b2683" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35a74422fbe4496abc46cf71d22af858" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 45895 } } peers { permanent_uuid: "b22f4448136b4b06945ad3ffa8b0231a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 39779 } } peers { permanent_uuid: "3c96b775eb0f489099f6169dd85b2683" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 42389 } } }
I20260620 19:10:31.256461 17871 tablet_replica.cc:442] T c76508d7446b4fceb93a26732aecbc4d P 35a74422fbe4496abc46cf71d22af858: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:10:31.269179 17860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:31.265843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.66:46229 (local address 127.9.155.67:42389)
0620 19:10:31.265976 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:31.265981 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:31.266184 (+ 203us) server_negotiation.cc:407] Connection header received
0620 19:10:31.266393 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:31.266396 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:31.266441 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:31.266505 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:10:31.266735 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:31.267236 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:31.268062 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:31.268356 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:31.268454 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:31.268745 (+ 291us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:10:31.268820 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:10:31.268874 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:31.269014 (+ 140us) server_negotiation.cc:299] Negotiation successful
0620 19:10:31.269069 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 19:10:31.269224 17913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:31.265734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.67:42389 (local address 127.9.155.66:46229)
0620 19:10:31.266099 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:31.266112 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:31.266221 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:31.266525 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:31.266528 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:31.266535 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 19:10:31.266635 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:31.266640 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:31.267334 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:31.267337 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:31.267959 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:31.267967 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:31.268459 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:31.268463 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:31.268562 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:31.268619 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 19:10:31.268893 (+ 274us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 19:10:31.268899 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 19:10:31.268962 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 19:10:31.269048 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":92,"threads_started":1}
I20260620 19:10:31.276077 17790 raft_consensus.cc:1261] T aff233c162034234ab68b9963000f08f P 3c96b775eb0f489099f6169dd85b2683 [term 1 FOLLOWER]: Refusing update from remote peer b22f4448136b4b06945ad3ffa8b0231a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:10:31.276180 17510 raft_consensus.cc:1261] T aff233c162034234ab68b9963000f08f P 35a74422fbe4496abc46cf71d22af858 [term 1 FOLLOWER]: Refusing update from remote peer b22f4448136b4b06945ad3ffa8b0231a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:10:31.276522 17874 consensus_queue.cc:1035] T aff233c162034234ab68b9963000f08f P b22f4448136b4b06945ad3ffa8b0231a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c96b775eb0f489099f6169dd85b2683" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 42389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:10:31.276656 17882 consensus_queue.cc:1035] T aff233c162034234ab68b9963000f08f P b22f4448136b4b06945ad3ffa8b0231a [LEADER]: Connected to new peer: Peer: permanent_uuid: "35a74422fbe4496abc46cf71d22af858" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 45895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:10:31.300482 17862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:31.296019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.67:36729 (local address 127.9.155.66:39779)
0620 19:10:31.296147 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:31.296152 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:31.296589 (+ 437us) server_negotiation.cc:407] Connection header received
0620 19:10:31.296798 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:31.296801 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:31.296854 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:31.296911 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:10:31.297654 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:31.298273 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:31.299176 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:31.299405 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:31.299492 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:31.299777 (+ 285us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:10:31.299865 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:10:31.299931 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:31.300187 (+ 256us) server_negotiation.cc:299] Negotiation successful
0620 19:10:31.300293 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 19:10:31.301216 17919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:31.295967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.66:39779 (local address 127.9.155.67:36729)
0620 19:10:31.296517 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:31.296532 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:31.296669 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:31.297157 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:31.297160 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:31.297168 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 19:10:31.297535 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:31.297544 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:31.298442 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:31.298444 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:31.299071 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:31.299078 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:31.299528 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:31.299532 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:31.299607 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:31.299653 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 19:10:31.299941 (+ 288us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 19:10:31.299947 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 19:10:31.300704 (+ 757us) client_negotiation.cc:240] Negotiation successful
0620 19:10:31.300792 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":129,"threads_started":1}
I20260620 19:10:31.328712 17352 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52100:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:10:31.328890 17352 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52100:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:10:31.331144 17352 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 42920bad573b462593296d6a1276cb32: Sending DeleteTablet for 3 replicas of tablet aff233c162034234ab68b9963000f08f
I20260620 19:10:31.331951 17490 tablet_service.cc:1514] Processing DeleteTablet for tablet aff233c162034234ab68b9963000f08f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:10:31 UTC) from {username='oryx', principal='oryx/127.9.155.126@KRBTEST.COM'} at 127.0.0.1:48450
I20260620 19:10:31.332072 17630 tablet_service.cc:1514] Processing DeleteTablet for tablet aff233c162034234ab68b9963000f08f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:10:31 UTC) from {username='oryx', principal='oryx/127.9.155.126@KRBTEST.COM'} at 127.0.0.1:48394
I20260620 19:10:31.332172 17888 tablet_replica.cc:330] T aff233c162034234ab68b9963000f08f P 35a74422fbe4496abc46cf71d22af858: stopping tablet replica
I20260620 19:10:31.332271 17887 tablet_replica.cc:330] T aff233c162034234ab68b9963000f08f P b22f4448136b4b06945ad3ffa8b0231a: stopping tablet replica
I20260620 19:10:31.332317 17888 raft_consensus.cc:2229] T aff233c162034234ab68b9963000f08f P 35a74422fbe4496abc46cf71d22af858 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:10:31.332373 17887 raft_consensus.cc:2229] T aff233c162034234ab68b9963000f08f P b22f4448136b4b06945ad3ffa8b0231a [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:10:31.332443 17888 raft_consensus.cc:2258] T aff233c162034234ab68b9963000f08f P 35a74422fbe4496abc46cf71d22af858 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:10:31.332567 17887 raft_consensus.cc:2258] T aff233c162034234ab68b9963000f08f P b22f4448136b4b06945ad3ffa8b0231a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:10:31.334640 17770 tablet_service.cc:1514] Processing DeleteTablet for tablet aff233c162034234ab68b9963000f08f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:10:31 UTC) from {username='oryx', principal='oryx/127.9.155.126@KRBTEST.COM'} at 127.0.0.1:57906
I20260620 19:10:31.334833 17889 tablet_replica.cc:330] T aff233c162034234ab68b9963000f08f P 3c96b775eb0f489099f6169dd85b2683: stopping tablet replica
I20260620 19:10:31.334955 17889 raft_consensus.cc:2229] T aff233c162034234ab68b9963000f08f P 3c96b775eb0f489099f6169dd85b2683 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:10:31.335095 17889 raft_consensus.cc:2258] T aff233c162034234ab68b9963000f08f P 3c96b775eb0f489099f6169dd85b2683 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:10:31.336297 17889 ts_tablet_manager.cc:1905] T aff233c162034234ab68b9963000f08f P 3c96b775eb0f489099f6169dd85b2683: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:10:31.336555 17888 ts_tablet_manager.cc:1905] T aff233c162034234ab68b9963000f08f P 35a74422fbe4496abc46cf71d22af858: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:10:31.339555 17889 ts_tablet_manager.cc:1918] T aff233c162034234ab68b9963000f08f P 3c96b775eb0f489099f6169dd85b2683: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 19:10:31.339643 17889 log.cc:1196] T aff233c162034234ab68b9963000f08f P 3c96b775eb0f489099f6169dd85b2683: Deleting WAL directory at /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/minicluster-data/ts-2/wal/wals/aff233c162034234ab68b9963000f08f
I20260620 19:10:31.339684 17888 ts_tablet_manager.cc:1918] T aff233c162034234ab68b9963000f08f P 35a74422fbe4496abc46cf71d22af858: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 19:10:31.339766 17888 log.cc:1196] T aff233c162034234ab68b9963000f08f P 35a74422fbe4496abc46cf71d22af858: Deleting WAL directory at /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/minicluster-data/ts-0/wal/wals/aff233c162034234ab68b9963000f08f
I20260620 19:10:31.339872 17889 ts_tablet_manager.cc:1939] T aff233c162034234ab68b9963000f08f P 3c96b775eb0f489099f6169dd85b2683: Deleting consensus metadata
I20260620 19:10:31.339988 17888 ts_tablet_manager.cc:1939] T aff233c162034234ab68b9963000f08f P 35a74422fbe4496abc46cf71d22af858: Deleting consensus metadata
I20260620 19:10:31.340793 17339 catalog_manager.cc:4841] TS 3c96b775eb0f489099f6169dd85b2683 (127.9.155.67:42389): tablet aff233c162034234ab68b9963000f08f (table test-table [id=f585d9db812a4612801666236d2ad51d]) successfully deleted
I20260620 19:10:31.341351 17887 ts_tablet_manager.cc:1905] T aff233c162034234ab68b9963000f08f P b22f4448136b4b06945ad3ffa8b0231a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:10:31.343677 17887 ts_tablet_manager.cc:1918] T aff233c162034234ab68b9963000f08f P b22f4448136b4b06945ad3ffa8b0231a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 19:10:31.343758 17887 log.cc:1196] T aff233c162034234ab68b9963000f08f P b22f4448136b4b06945ad3ffa8b0231a: Deleting WAL directory at /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/minicluster-data/ts-1/wal/wals/aff233c162034234ab68b9963000f08f
I20260620 19:10:31.343950 17887 ts_tablet_manager.cc:1939] T aff233c162034234ab68b9963000f08f P b22f4448136b4b06945ad3ffa8b0231a: Deleting consensus metadata
I20260620 19:10:31.345021 17337 catalog_manager.cc:4841] TS 35a74422fbe4496abc46cf71d22af858 (127.9.155.65:45895): tablet aff233c162034234ab68b9963000f08f (table test-table [id=f585d9db812a4612801666236d2ad51d]) successfully deleted
I20260620 19:10:31.345108 17340 catalog_manager.cc:4841] TS b22f4448136b4b06945ad3ffa8b0231a (127.9.155.66:39779): tablet aff233c162034234ab68b9963000f08f (table test-table [id=f585d9db812a4612801666236d2ad51d]) successfully deleted
Jun 20 19:10:31 dist-test-slave-n3fx krb5kdc[17308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 19:10:31 dist-test-slave-n3fx krb5kdc[17308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:10:31 dist-test-slave-n3fx krb5kdc[17308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
W20260620 19:10:31.365065 17402 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:10:31.357051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52124 (local address 127.9.155.126:39669)
0620 19:10:31.357205 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:31.357210 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:31.357441 (+ 231us) server_negotiation.cc:407] Connection header received
0620 19:10:31.357600 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:31.357602 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:31.357651 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:31.357723 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:31.358699 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:31.359239 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:31.360604 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:31.360859 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:31.360952 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:31.364930 (+ 3978us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52124: BlockingRecv error: recv got EOF from 127.0.0.1:52124 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 20 19:10:31 dist-test-slave-n3fx krb5kdc[17308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM
I20260620 19:10:31.375674 17402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:31.366783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52138 (local address 127.9.155.126:39669)
0620 19:10:31.366906 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:31.366910 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:31.367014 (+ 104us) server_negotiation.cc:407] Connection header received
0620 19:10:31.367117 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:31.367121 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:31.367168 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:31.367255 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:31.367804 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:31.368660 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:31.369478 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:31.369804 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:31.369919 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:31.372304 (+ 2385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:31.372308 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:31.372311 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:31.372345 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:31.373956 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:31.374431 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:31.374435 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:31.374437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:31.374520 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:31.374780 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:31.374798 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:31.374800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:31.375001 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:31.375124 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:31.375378 (+ 254us) server_negotiation.cc:299] Negotiation successful
0620 19:10:31.375498 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 19:10:31.378480 17352 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52138:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:10:31.378927 17352 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:10:31.383692 17490 tablet_service.cc:1467] Processing CreateTablet for tablet ce683902b8dc4f1da9a1b55c89acc556 (DEFAULT_TABLE table=test-table [id=b4217a52f4c642f18a8ff6d0709fbada]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:10:31.383883 17770 tablet_service.cc:1467] Processing CreateTablet for tablet ce683902b8dc4f1da9a1b55c89acc556 (DEFAULT_TABLE table=test-table [id=b4217a52f4c642f18a8ff6d0709fbada]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:10:31.383924 17630 tablet_service.cc:1467] Processing CreateTablet for tablet ce683902b8dc4f1da9a1b55c89acc556 (DEFAULT_TABLE table=test-table [id=b4217a52f4c642f18a8ff6d0709fbada]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:10:31.384057 17490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce683902b8dc4f1da9a1b55c89acc556. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:31.384125 17770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce683902b8dc4f1da9a1b55c89acc556. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:31.384274 17630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce683902b8dc4f1da9a1b55c89acc556. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:31.386554 17864 tablet_bootstrap.cc:492] T ce683902b8dc4f1da9a1b55c89acc556 P 35a74422fbe4496abc46cf71d22af858: Bootstrap starting.
I20260620 19:10:31.386554 17865 tablet_bootstrap.cc:492] T ce683902b8dc4f1da9a1b55c89acc556 P 3c96b775eb0f489099f6169dd85b2683: Bootstrap starting.
I20260620 19:10:31.386612 17866 tablet_bootstrap.cc:492] T ce683902b8dc4f1da9a1b55c89acc556 P b22f4448136b4b06945ad3ffa8b0231a: Bootstrap starting.
I20260620 19:10:31.387385 17866 tablet_bootstrap.cc:654] T ce683902b8dc4f1da9a1b55c89acc556 P b22f4448136b4b06945ad3ffa8b0231a: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:31.387382 17864 tablet_bootstrap.cc:654] T ce683902b8dc4f1da9a1b55c89acc556 P 35a74422fbe4496abc46cf71d22af858: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:31.387392 17865 tablet_bootstrap.cc:654] T ce683902b8dc4f1da9a1b55c89acc556 P 3c96b775eb0f489099f6169dd85b2683: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:31.388957 17866 tablet_bootstrap.cc:492] T ce683902b8dc4f1da9a1b55c89acc556 P b22f4448136b4b06945ad3ffa8b0231a: No bootstrap required, opened a new log
I20260620 19:10:31.388957 17865 tablet_bootstrap.cc:492] T ce683902b8dc4f1da9a1b55c89acc556 P 3c96b775eb0f489099f6169dd85b2683: No bootstrap required, opened a new log
I20260620 19:10:31.389040 17866 ts_tablet_manager.cc:1397] T ce683902b8dc4f1da9a1b55c89acc556 P b22f4448136b4b06945ad3ffa8b0231a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:10:31.389040 17865 ts_tablet_manager.cc:1397] T ce683902b8dc4f1da9a1b55c89acc556 P 3c96b775eb0f489099f6169dd85b2683: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 19:10:31.389429 17865 raft_consensus.cc:348] T ce683902b8dc4f1da9a1b55c89acc556 P 3c96b775eb0f489099f6169dd85b2683 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35a74422fbe4496abc46cf71d22af858" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 45895 } } peers { permanent_uuid: "b22f4448136b4b06945ad3ffa8b0231a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 39779 } } peers { permanent_uuid: "3c96b775eb0f489099f6169dd85b2683" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 42389 } }
I20260620 19:10:31.389430 17866 raft_consensus.cc:348] T ce683902b8dc4f1da9a1b55c89acc556 P b22f4448136b4b06945ad3ffa8b0231a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35a74422fbe4496abc46cf71d22af858" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 45895 } } peers { permanent_uuid: "b22f4448136b4b06945ad3ffa8b0231a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 39779 } } peers { permanent_uuid: "3c96b775eb0f489099f6169dd85b2683" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 42389 } }
I20260620 19:10:31.389537 17866 raft_consensus.cc:374] T ce683902b8dc4f1da9a1b55c89acc556 P b22f4448136b4b06945ad3ffa8b0231a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:31.389537 17865 raft_consensus.cc:374] T ce683902b8dc4f1da9a1b55c89acc556 P 3c96b775eb0f489099f6169dd85b2683 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:31.389539 17864 tablet_bootstrap.cc:492] T ce683902b8dc4f1da9a1b55c89acc556 P 35a74422fbe4496abc46cf71d22af858: No bootstrap required, opened a new log
I20260620 19:10:31.389569 17866 raft_consensus.cc:729] T ce683902b8dc4f1da9a1b55c89acc556 P b22f4448136b4b06945ad3ffa8b0231a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22f4448136b4b06945ad3ffa8b0231a, State: Initialized, Role: FOLLOWER
I20260620 19:10:31.389569 17865 raft_consensus.cc:729] T ce683902b8dc4f1da9a1b55c89acc556 P 3c96b775eb0f489099f6169dd85b2683 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c96b775eb0f489099f6169dd85b2683, State: Initialized, Role: FOLLOWER
I20260620 19:10:31.389604 17864 ts_tablet_manager.cc:1397] T ce683902b8dc4f1da9a1b55c89acc556 P 35a74422fbe4496abc46cf71d22af858: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 19:10:31.389689 17865 consensus_queue.cc:260] T ce683902b8dc4f1da9a1b55c89acc556 P 3c96b775eb0f489099f6169dd85b2683 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35a74422fbe4496abc46cf71d22af858" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 45895 } } peers { permanent_uuid: "b22f4448136b4b06945ad3ffa8b0231a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 39779 } } peers { permanent_uuid: "3c96b775eb0f489099f6169dd85b2683" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 42389 } }
I20260620 19:10:31.389689 17866 consensus_queue.cc:260] T ce683902b8dc4f1da9a1b55c89acc556 P b22f4448136b4b06945ad3ffa8b0231a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35a74422fbe4496abc46cf71d22af858" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 45895 } } peers { permanent_uuid: "b22f4448136b4b06945ad3ffa8b0231a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 39779 } } peers { permanent_uuid: "3c96b775eb0f489099f6169dd85b2683" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 42389 } }
I20260620 19:10:31.389947 17866 ts_tablet_manager.cc:1428] T ce683902b8dc4f1da9a1b55c89acc556 P b22f4448136b4b06945ad3ffa8b0231a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 19:10:31.389957 17865 ts_tablet_manager.cc:1428] T ce683902b8dc4f1da9a1b55c89acc556 P 3c96b775eb0f489099f6169dd85b2683: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:10:31.390110 17864 raft_consensus.cc:348] T ce683902b8dc4f1da9a1b55c89acc556 P 35a74422fbe4496abc46cf71d22af858 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35a74422fbe4496abc46cf71d22af858" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 45895 } } peers { permanent_uuid: "b22f4448136b4b06945ad3ffa8b0231a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 39779 } } peers { permanent_uuid: "3c96b775eb0f489099f6169dd85b2683" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 42389 } }
I20260620 19:10:31.390221 17864 raft_consensus.cc:374] T ce683902b8dc4f1da9a1b55c89acc556 P 35a74422fbe4496abc46cf71d22af858 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:31.390249 17864 raft_consensus.cc:729] T ce683902b8dc4f1da9a1b55c89acc556 P 35a74422fbe4496abc46cf71d22af858 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35a74422fbe4496abc46cf71d22af858, State: Initialized, Role: FOLLOWER
I20260620 19:10:31.390358 17864 consensus_queue.cc:260] T ce683902b8dc4f1da9a1b55c89acc556 P 35a74422fbe4496abc46cf71d22af858 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35a74422fbe4496abc46cf71d22af858" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 45895 } } peers { permanent_uuid: "b22f4448136b4b06945ad3ffa8b0231a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 39779 } } peers { permanent_uuid: "3c96b775eb0f489099f6169dd85b2683" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 42389 } }
I20260620 19:10:31.390623 17864 ts_tablet_manager.cc:1428] T ce683902b8dc4f1da9a1b55c89acc556 P 35a74422fbe4496abc46cf71d22af858: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:10:31.717736 17874 raft_consensus.cc:482] T ce683902b8dc4f1da9a1b55c89acc556 P b22f4448136b4b06945ad3ffa8b0231a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:10:31.717875 17874 raft_consensus.cc:504] T ce683902b8dc4f1da9a1b55c89acc556 P b22f4448136b4b06945ad3ffa8b0231a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35a74422fbe4496abc46cf71d22af858" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 45895 } } peers { permanent_uuid: "b22f4448136b4b06945ad3ffa8b0231a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 39779 } } peers { permanent_uuid: "3c96b775eb0f489099f6169dd85b2683" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 42389 } }
I20260620 19:10:31.718261 17874 leader_election.cc:288] T ce683902b8dc4f1da9a1b55c89acc556 P b22f4448136b4b06945ad3ffa8b0231a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 35a74422fbe4496abc46cf71d22af858 (127.9.155.65:45895), 3c96b775eb0f489099f6169dd85b2683 (127.9.155.67:42389)
I20260620 19:10:31.718775 17510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce683902b8dc4f1da9a1b55c89acc556" candidate_uuid: "b22f4448136b4b06945ad3ffa8b0231a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35a74422fbe4496abc46cf71d22af858" is_pre_election: true
I20260620 19:10:31.718775 17790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce683902b8dc4f1da9a1b55c89acc556" candidate_uuid: "b22f4448136b4b06945ad3ffa8b0231a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c96b775eb0f489099f6169dd85b2683" is_pre_election: true
I20260620 19:10:31.718940 17510 raft_consensus.cc:2454] T ce683902b8dc4f1da9a1b55c89acc556 P 35a74422fbe4496abc46cf71d22af858 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22f4448136b4b06945ad3ffa8b0231a in term 0.
I20260620 19:10:31.718950 17790 raft_consensus.cc:2454] T ce683902b8dc4f1da9a1b55c89acc556 P 3c96b775eb0f489099f6169dd85b2683 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22f4448136b4b06945ad3ffa8b0231a in term 0.
I20260620 19:10:31.719283 17583 leader_election.cc:302] T ce683902b8dc4f1da9a1b55c89acc556 P b22f4448136b4b06945ad3ffa8b0231a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 35a74422fbe4496abc46cf71d22af858, 3c96b775eb0f489099f6169dd85b2683, b22f4448136b4b06945ad3ffa8b0231a; no voters:
I20260620 19:10:31.719465 17874 raft_consensus.cc:2781] T ce683902b8dc4f1da9a1b55c89acc556 P b22f4448136b4b06945ad3ffa8b0231a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:10:31.719529 17874 raft_consensus.cc:482] T ce683902b8dc4f1da9a1b55c89acc556 P b22f4448136b4b06945ad3ffa8b0231a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:10:31.719570 17874 raft_consensus.cc:3037] T ce683902b8dc4f1da9a1b55c89acc556 P b22f4448136b4b06945ad3ffa8b0231a [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:31.720628 17874 raft_consensus.cc:504] T ce683902b8dc4f1da9a1b55c89acc556 P b22f4448136b4b06945ad3ffa8b0231a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35a74422fbe4496abc46cf71d22af858" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 45895 } } peers { permanent_uuid: "b22f4448136b4b06945ad3ffa8b0231a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 39779 } } peers { permanent_uuid: "3c96b775eb0f489099f6169dd85b2683" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 42389 } }
I20260620 19:10:31.720953 17874 leader_election.cc:288] T ce683902b8dc4f1da9a1b55c89acc556 P b22f4448136b4b06945ad3ffa8b0231a [CANDIDATE]: Term 1 election: Requested vote from peers 35a74422fbe4496abc46cf71d22af858 (127.9.155.65:45895), 3c96b775eb0f489099f6169dd85b2683 (127.9.155.67:42389)
I20260620 19:10:31.721365 17790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce683902b8dc4f1da9a1b55c89acc556" candidate_uuid: "b22f4448136b4b06945ad3ffa8b0231a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c96b775eb0f489099f6169dd85b2683"
I20260620 19:10:31.721349 17510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce683902b8dc4f1da9a1b55c89acc556" candidate_uuid: "b22f4448136b4b06945ad3ffa8b0231a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35a74422fbe4496abc46cf71d22af858"
I20260620 19:10:31.721483 17790 raft_consensus.cc:3037] T ce683902b8dc4f1da9a1b55c89acc556 P 3c96b775eb0f489099f6169dd85b2683 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:31.721503 17510 raft_consensus.cc:3037] T ce683902b8dc4f1da9a1b55c89acc556 P 35a74422fbe4496abc46cf71d22af858 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:31.722604 17510 raft_consensus.cc:2454] T ce683902b8dc4f1da9a1b55c89acc556 P 35a74422fbe4496abc46cf71d22af858 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22f4448136b4b06945ad3ffa8b0231a in term 1.
I20260620 19:10:31.722604 17790 raft_consensus.cc:2454] T ce683902b8dc4f1da9a1b55c89acc556 P 3c96b775eb0f489099f6169dd85b2683 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22f4448136b4b06945ad3ffa8b0231a in term 1.
I20260620 19:10:31.722894 17581 leader_election.cc:302] T ce683902b8dc4f1da9a1b55c89acc556 P b22f4448136b4b06945ad3ffa8b0231a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 35a74422fbe4496abc46cf71d22af858, 3c96b775eb0f489099f6169dd85b2683, b22f4448136b4b06945ad3ffa8b0231a; no voters:
I20260620 19:10:31.723081 17874 raft_consensus.cc:2781] T ce683902b8dc4f1da9a1b55c89acc556 P b22f4448136b4b06945ad3ffa8b0231a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:10:31.723187 17874 raft_consensus.cc:686] T ce683902b8dc4f1da9a1b55c89acc556 P b22f4448136b4b06945ad3ffa8b0231a [term 1 LEADER]: Becoming Leader. State: Replica: b22f4448136b4b06945ad3ffa8b0231a, State: Running, Role: LEADER
I20260620 19:10:31.723325 17874 consensus_queue.cc:237] T ce683902b8dc4f1da9a1b55c89acc556 P b22f4448136b4b06945ad3ffa8b0231a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35a74422fbe4496abc46cf71d22af858" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 45895 } } peers { permanent_uuid: "b22f4448136b4b06945ad3ffa8b0231a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 39779 } } peers { permanent_uuid: "3c96b775eb0f489099f6169dd85b2683" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 42389 } }
I20260620 19:10:31.725054 17350 catalog_manager.cc:5495] T ce683902b8dc4f1da9a1b55c89acc556 P b22f4448136b4b06945ad3ffa8b0231a reported cstate change: term changed from 0 to 1, leader changed from <none> to b22f4448136b4b06945ad3ffa8b0231a (127.9.155.66). New cstate: current_term: 1 leader_uuid: "b22f4448136b4b06945ad3ffa8b0231a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35a74422fbe4496abc46cf71d22af858" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 45895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b22f4448136b4b06945ad3ffa8b0231a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 39779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c96b775eb0f489099f6169dd85b2683" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 42389 } health_report { overall_health: UNKNOWN } } }
I20260620 19:10:31.758673 17862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:31.755549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48418 (local address 127.9.155.66:39779)
0620 19:10:31.755672 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:31.755675 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:31.755688 (+ 13us) server_negotiation.cc:407] Connection header received
0620 19:10:31.755789 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:31.755791 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:31.755838 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:31.755899 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:10:31.756102 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:31.756669 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:31.757547 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:31.757772 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:31.757864 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:31.758205 (+ 341us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:10:31.758287 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:10:31.758351 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:31.758476 (+ 125us) server_negotiation.cc:299] Negotiation successful
0620 19:10:31.758530 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 19:10:31.760759 17790 raft_consensus.cc:1261] T ce683902b8dc4f1da9a1b55c89acc556 P 3c96b775eb0f489099f6169dd85b2683 [term 1 FOLLOWER]: Refusing update from remote peer b22f4448136b4b06945ad3ffa8b0231a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:10:31.760759 17510 raft_consensus.cc:1261] T ce683902b8dc4f1da9a1b55c89acc556 P 35a74422fbe4496abc46cf71d22af858 [term 1 FOLLOWER]: Refusing update from remote peer b22f4448136b4b06945ad3ffa8b0231a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:10:31.761241 17882 consensus_queue.cc:1035] T ce683902b8dc4f1da9a1b55c89acc556 P b22f4448136b4b06945ad3ffa8b0231a [LEADER]: Connected to new peer: Peer: permanent_uuid: "35a74422fbe4496abc46cf71d22af858" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 45895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:10:31.761408 17874 consensus_queue.cc:1035] T ce683902b8dc4f1da9a1b55c89acc556 P b22f4448136b4b06945ad3ffa8b0231a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c96b775eb0f489099f6169dd85b2683" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 42389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:10:31.768555 17351 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52138:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:10:31.768688 17351 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52138:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:10:31.770426 17351 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 42920bad573b462593296d6a1276cb32: Sending DeleteTablet for 3 replicas of tablet ce683902b8dc4f1da9a1b55c89acc556
I20260620 19:10:31.770920 17490 tablet_service.cc:1514] Processing DeleteTablet for tablet ce683902b8dc4f1da9a1b55c89acc556 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:10:31 UTC) from {username='oryx', principal='oryx/127.9.155.126@KRBTEST.COM'} at 127.0.0.1:48450
I20260620 19:10:31.770939 17630 tablet_service.cc:1514] Processing DeleteTablet for tablet ce683902b8dc4f1da9a1b55c89acc556 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:10:31 UTC) from {username='oryx', principal='oryx/127.9.155.126@KRBTEST.COM'} at 127.0.0.1:48394
I20260620 19:10:31.771118 17887 tablet_replica.cc:330] T ce683902b8dc4f1da9a1b55c89acc556 P b22f4448136b4b06945ad3ffa8b0231a: stopping tablet replica
I20260620 19:10:31.771112 17770 tablet_service.cc:1514] Processing DeleteTablet for tablet ce683902b8dc4f1da9a1b55c89acc556 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:10:31 UTC) from {username='oryx', principal='oryx/127.9.155.126@KRBTEST.COM'} at 127.0.0.1:57906
I20260620 19:10:31.771227 17887 raft_consensus.cc:2229] T ce683902b8dc4f1da9a1b55c89acc556 P b22f4448136b4b06945ad3ffa8b0231a [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:10:31.771270 17888 tablet_replica.cc:330] T ce683902b8dc4f1da9a1b55c89acc556 P 35a74422fbe4496abc46cf71d22af858: stopping tablet replica
I20260620 19:10:31.771366 17888 raft_consensus.cc:2229] T ce683902b8dc4f1da9a1b55c89acc556 P 35a74422fbe4496abc46cf71d22af858 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:10:31.771400 17887 raft_consensus.cc:2258] T ce683902b8dc4f1da9a1b55c89acc556 P b22f4448136b4b06945ad3ffa8b0231a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:10:31.771477 17888 raft_consensus.cc:2258] T ce683902b8dc4f1da9a1b55c89acc556 P 35a74422fbe4496abc46cf71d22af858 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:10:31.771628 17889 tablet_replica.cc:330] T ce683902b8dc4f1da9a1b55c89acc556 P 3c96b775eb0f489099f6169dd85b2683: stopping tablet replica
I20260620 19:10:31.771729 17889 raft_consensus.cc:2229] T ce683902b8dc4f1da9a1b55c89acc556 P 3c96b775eb0f489099f6169dd85b2683 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:10:31.771734 17351 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52138:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 19:10:31.771845 17889 raft_consensus.cc:2258] T ce683902b8dc4f1da9a1b55c89acc556 P 3c96b775eb0f489099f6169dd85b2683 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 19:10:31.772100 17351 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:10:31.772497 17889 ts_tablet_manager.cc:1905] T ce683902b8dc4f1da9a1b55c89acc556 P 3c96b775eb0f489099f6169dd85b2683: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:10:31.772497 17888 ts_tablet_manager.cc:1905] T ce683902b8dc4f1da9a1b55c89acc556 P 35a74422fbe4496abc46cf71d22af858: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:10:31.772668 17887 ts_tablet_manager.cc:1905] T ce683902b8dc4f1da9a1b55c89acc556 P b22f4448136b4b06945ad3ffa8b0231a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:10:31.774736 17888 ts_tablet_manager.cc:1918] T ce683902b8dc4f1da9a1b55c89acc556 P 35a74422fbe4496abc46cf71d22af858: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:10:31.774736 17889 ts_tablet_manager.cc:1918] T ce683902b8dc4f1da9a1b55c89acc556 P 3c96b775eb0f489099f6169dd85b2683: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:10:31.774801 17889 log.cc:1196] T ce683902b8dc4f1da9a1b55c89acc556 P 3c96b775eb0f489099f6169dd85b2683: Deleting WAL directory at /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/minicluster-data/ts-2/wal/wals/ce683902b8dc4f1da9a1b55c89acc556
I20260620 19:10:31.774801 17888 log.cc:1196] T ce683902b8dc4f1da9a1b55c89acc556 P 35a74422fbe4496abc46cf71d22af858: Deleting WAL directory at /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/minicluster-data/ts-0/wal/wals/ce683902b8dc4f1da9a1b55c89acc556
I20260620 19:10:31.775031 17888 ts_tablet_manager.cc:1939] T ce683902b8dc4f1da9a1b55c89acc556 P 35a74422fbe4496abc46cf71d22af858: Deleting consensus metadata
I20260620 19:10:31.775041 17889 ts_tablet_manager.cc:1939] T ce683902b8dc4f1da9a1b55c89acc556 P 3c96b775eb0f489099f6169dd85b2683: Deleting consensus metadata
I20260620 19:10:31.775031 17887 ts_tablet_manager.cc:1918] T ce683902b8dc4f1da9a1b55c89acc556 P b22f4448136b4b06945ad3ffa8b0231a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:10:31.775264 17887 log.cc:1196] T ce683902b8dc4f1da9a1b55c89acc556 P b22f4448136b4b06945ad3ffa8b0231a: Deleting WAL directory at /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/minicluster-data/ts-1/wal/wals/ce683902b8dc4f1da9a1b55c89acc556
I20260620 19:10:31.775527 17887 ts_tablet_manager.cc:1939] T ce683902b8dc4f1da9a1b55c89acc556 P b22f4448136b4b06945ad3ffa8b0231a: Deleting consensus metadata
I20260620 19:10:31.775667 17339 catalog_manager.cc:4841] TS 3c96b775eb0f489099f6169dd85b2683 (127.9.155.67:42389): tablet ce683902b8dc4f1da9a1b55c89acc556 (table test-table [id=b4217a52f4c642f18a8ff6d0709fbada]) successfully deleted
I20260620 19:10:31.775753 17337 catalog_manager.cc:4841] TS 35a74422fbe4496abc46cf71d22af858 (127.9.155.65:45895): tablet ce683902b8dc4f1da9a1b55c89acc556 (table test-table [id=b4217a52f4c642f18a8ff6d0709fbada]) successfully deleted
I20260620 19:10:31.776140 17340 catalog_manager.cc:4841] TS b22f4448136b4b06945ad3ffa8b0231a (127.9.155.66:39779): tablet ce683902b8dc4f1da9a1b55c89acc556 (table test-table [id=b4217a52f4c642f18a8ff6d0709fbada]) successfully deleted
I20260620 19:10:31.776984 17630 tablet_service.cc:1467] Processing CreateTablet for tablet c166820f9a314c0da2914de636252c0d (DEFAULT_TABLE table=test-table [id=51f5e82abe4242e795eab664be95bda6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:10:31.777068 17770 tablet_service.cc:1467] Processing CreateTablet for tablet c166820f9a314c0da2914de636252c0d (DEFAULT_TABLE table=test-table [id=51f5e82abe4242e795eab664be95bda6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:10:31.777227 17630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c166820f9a314c0da2914de636252c0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:31.777318 17770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c166820f9a314c0da2914de636252c0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:31.779040 17866 tablet_bootstrap.cc:492] T c166820f9a314c0da2914de636252c0d P b22f4448136b4b06945ad3ffa8b0231a: Bootstrap starting.
I20260620 19:10:31.779347 17490 tablet_service.cc:1467] Processing CreateTablet for tablet c166820f9a314c0da2914de636252c0d (DEFAULT_TABLE table=test-table [id=51f5e82abe4242e795eab664be95bda6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:10:31.779587 17490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c166820f9a314c0da2914de636252c0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:31.779848 17866 tablet_bootstrap.cc:654] T c166820f9a314c0da2914de636252c0d P b22f4448136b4b06945ad3ffa8b0231a: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:31.780367 17865 tablet_bootstrap.cc:492] T c166820f9a314c0da2914de636252c0d P 3c96b775eb0f489099f6169dd85b2683: Bootstrap starting.
I20260620 19:10:31.780944 17866 tablet_bootstrap.cc:492] T c166820f9a314c0da2914de636252c0d P b22f4448136b4b06945ad3ffa8b0231a: No bootstrap required, opened a new log
I20260620 19:10:31.781023 17866 ts_tablet_manager.cc:1397] T c166820f9a314c0da2914de636252c0d P b22f4448136b4b06945ad3ffa8b0231a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 19:10:31.781143 17865 tablet_bootstrap.cc:654] T c166820f9a314c0da2914de636252c0d P 3c96b775eb0f489099f6169dd85b2683: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:31.781373 17866 raft_consensus.cc:348] T c166820f9a314c0da2914de636252c0d P b22f4448136b4b06945ad3ffa8b0231a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35a74422fbe4496abc46cf71d22af858" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 45895 } } peers { permanent_uuid: "b22f4448136b4b06945ad3ffa8b0231a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 39779 } } peers { permanent_uuid: "3c96b775eb0f489099f6169dd85b2683" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 42389 } }
I20260620 19:10:31.781466 17866 raft_consensus.cc:374] T c166820f9a314c0da2914de636252c0d P b22f4448136b4b06945ad3ffa8b0231a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:31.781510 17866 raft_consensus.cc:729] T c166820f9a314c0da2914de636252c0d P b22f4448136b4b06945ad3ffa8b0231a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22f4448136b4b06945ad3ffa8b0231a, State: Initialized, Role: FOLLOWER
I20260620 19:10:31.781627 17866 consensus_queue.cc:260] T c166820f9a314c0da2914de636252c0d P b22f4448136b4b06945ad3ffa8b0231a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35a74422fbe4496abc46cf71d22af858" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 45895 } } peers { permanent_uuid: "b22f4448136b4b06945ad3ffa8b0231a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 39779 } } peers { permanent_uuid: "3c96b775eb0f489099f6169dd85b2683" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 42389 } }
I20260620 19:10:31.781847 17864 tablet_bootstrap.cc:492] T c166820f9a314c0da2914de636252c0d P 35a74422fbe4496abc46cf71d22af858: Bootstrap starting.
I20260620 19:10:31.781901 17866 ts_tablet_manager.cc:1428] T c166820f9a314c0da2914de636252c0d P b22f4448136b4b06945ad3ffa8b0231a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 19:10:31.782091 17865 tablet_bootstrap.cc:492] T c166820f9a314c0da2914de636252c0d P 3c96b775eb0f489099f6169dd85b2683: No bootstrap required, opened a new log
I20260620 19:10:31.782176 17865 ts_tablet_manager.cc:1397] T c166820f9a314c0da2914de636252c0d P 3c96b775eb0f489099f6169dd85b2683: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:10:31.782541 17865 raft_consensus.cc:348] T c166820f9a314c0da2914de636252c0d P 3c96b775eb0f489099f6169dd85b2683 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35a74422fbe4496abc46cf71d22af858" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 45895 } } peers { permanent_uuid: "b22f4448136b4b06945ad3ffa8b0231a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 39779 } } peers { permanent_uuid: "3c96b775eb0f489099f6169dd85b2683" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 42389 } }
I20260620 19:10:31.782652 17865 raft_consensus.cc:374] T c166820f9a314c0da2914de636252c0d P 3c96b775eb0f489099f6169dd85b2683 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:31.782692 17865 raft_consensus.cc:729] T c166820f9a314c0da2914de636252c0d P 3c96b775eb0f489099f6169dd85b2683 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c96b775eb0f489099f6169dd85b2683, State: Initialized, Role: FOLLOWER
I20260620 19:10:31.782704 17864 tablet_bootstrap.cc:654] T c166820f9a314c0da2914de636252c0d P 35a74422fbe4496abc46cf71d22af858: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:31.782814 17865 consensus_queue.cc:260] T c166820f9a314c0da2914de636252c0d P 3c96b775eb0f489099f6169dd85b2683 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35a74422fbe4496abc46cf71d22af858" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 45895 } } peers { permanent_uuid: "b22f4448136b4b06945ad3ffa8b0231a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 39779 } } peers { permanent_uuid: "3c96b775eb0f489099f6169dd85b2683" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 42389 } }
I20260620 19:10:31.783082 17865 ts_tablet_manager.cc:1428] T c166820f9a314c0da2914de636252c0d P 3c96b775eb0f489099f6169dd85b2683: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 19:10:31.784267 17864 tablet_bootstrap.cc:492] T c166820f9a314c0da2914de636252c0d P 35a74422fbe4496abc46cf71d22af858: No bootstrap required, opened a new log
I20260620 19:10:31.784353 17864 ts_tablet_manager.cc:1397] T c166820f9a314c0da2914de636252c0d P 35a74422fbe4496abc46cf71d22af858: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 19:10:31.784876 17864 raft_consensus.cc:348] T c166820f9a314c0da2914de636252c0d P 35a74422fbe4496abc46cf71d22af858 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35a74422fbe4496abc46cf71d22af858" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 45895 } } peers { permanent_uuid: "b22f4448136b4b06945ad3ffa8b0231a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 39779 } } peers { permanent_uuid: "3c96b775eb0f489099f6169dd85b2683" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 42389 } }
I20260620 19:10:31.784998 17864 raft_consensus.cc:374] T c166820f9a314c0da2914de636252c0d P 35a74422fbe4496abc46cf71d22af858 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:31.785029 17864 raft_consensus.cc:729] T c166820f9a314c0da2914de636252c0d P 35a74422fbe4496abc46cf71d22af858 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35a74422fbe4496abc46cf71d22af858, State: Initialized, Role: FOLLOWER
I20260620 19:10:31.785142 17864 consensus_queue.cc:260] T c166820f9a314c0da2914de636252c0d P 35a74422fbe4496abc46cf71d22af858 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35a74422fbe4496abc46cf71d22af858" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 45895 } } peers { permanent_uuid: "b22f4448136b4b06945ad3ffa8b0231a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 39779 } } peers { permanent_uuid: "3c96b775eb0f489099f6169dd85b2683" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 42389 } }
I20260620 19:10:31.785399 17864 ts_tablet_manager.cc:1428] T c166820f9a314c0da2914de636252c0d P 35a74422fbe4496abc46cf71d22af858: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 19:10:31.932541 17874 raft_consensus.cc:482] T c166820f9a314c0da2914de636252c0d P b22f4448136b4b06945ad3ffa8b0231a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:10:31.932683 17874 raft_consensus.cc:504] T c166820f9a314c0da2914de636252c0d P b22f4448136b4b06945ad3ffa8b0231a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35a74422fbe4496abc46cf71d22af858" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 45895 } } peers { permanent_uuid: "b22f4448136b4b06945ad3ffa8b0231a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 39779 } } peers { permanent_uuid: "3c96b775eb0f489099f6169dd85b2683" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 42389 } }
I20260620 19:10:31.933087 17874 leader_election.cc:288] T c166820f9a314c0da2914de636252c0d P b22f4448136b4b06945ad3ffa8b0231a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 35a74422fbe4496abc46cf71d22af858 (127.9.155.65:45895), 3c96b775eb0f489099f6169dd85b2683 (127.9.155.67:42389)
I20260620 19:10:31.933600 17790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c166820f9a314c0da2914de636252c0d" candidate_uuid: "b22f4448136b4b06945ad3ffa8b0231a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c96b775eb0f489099f6169dd85b2683" is_pre_election: true
I20260620 19:10:31.933635 17510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c166820f9a314c0da2914de636252c0d" candidate_uuid: "b22f4448136b4b06945ad3ffa8b0231a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35a74422fbe4496abc46cf71d22af858" is_pre_election: true
I20260620 19:10:31.933779 17790 raft_consensus.cc:2454] T c166820f9a314c0da2914de636252c0d P 3c96b775eb0f489099f6169dd85b2683 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22f4448136b4b06945ad3ffa8b0231a in term 0.
I20260620 19:10:31.933779 17510 raft_consensus.cc:2454] T c166820f9a314c0da2914de636252c0d P 35a74422fbe4496abc46cf71d22af858 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22f4448136b4b06945ad3ffa8b0231a in term 0.
I20260620 19:10:31.934159 17581 leader_election.cc:302] T c166820f9a314c0da2914de636252c0d P b22f4448136b4b06945ad3ffa8b0231a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 35a74422fbe4496abc46cf71d22af858, 3c96b775eb0f489099f6169dd85b2683, b22f4448136b4b06945ad3ffa8b0231a; no voters:
I20260620 19:10:31.934370 17874 raft_consensus.cc:2781] T c166820f9a314c0da2914de636252c0d P b22f4448136b4b06945ad3ffa8b0231a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:10:31.934430 17874 raft_consensus.cc:482] T c166820f9a314c0da2914de636252c0d P b22f4448136b4b06945ad3ffa8b0231a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:10:31.934455 17874 raft_consensus.cc:3037] T c166820f9a314c0da2914de636252c0d P b22f4448136b4b06945ad3ffa8b0231a [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:31.935506 17874 raft_consensus.cc:504] T c166820f9a314c0da2914de636252c0d P b22f4448136b4b06945ad3ffa8b0231a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35a74422fbe4496abc46cf71d22af858" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 45895 } } peers { permanent_uuid: "b22f4448136b4b06945ad3ffa8b0231a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 39779 } } peers { permanent_uuid: "3c96b775eb0f489099f6169dd85b2683" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 42389 } }
I20260620 19:10:31.935951 17874 leader_election.cc:288] T c166820f9a314c0da2914de636252c0d P b22f4448136b4b06945ad3ffa8b0231a [CANDIDATE]: Term 1 election: Requested vote from peers 35a74422fbe4496abc46cf71d22af858 (127.9.155.65:45895), 3c96b775eb0f489099f6169dd85b2683 (127.9.155.67:42389)
I20260620 19:10:31.936450 17510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c166820f9a314c0da2914de636252c0d" candidate_uuid: "b22f4448136b4b06945ad3ffa8b0231a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35a74422fbe4496abc46cf71d22af858"
I20260620 19:10:31.936434 17790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c166820f9a314c0da2914de636252c0d" candidate_uuid: "b22f4448136b4b06945ad3ffa8b0231a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c96b775eb0f489099f6169dd85b2683"
I20260620 19:10:31.936581 17510 raft_consensus.cc:3037] T c166820f9a314c0da2914de636252c0d P 35a74422fbe4496abc46cf71d22af858 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:31.936581 17790 raft_consensus.cc:3037] T c166820f9a314c0da2914de636252c0d P 3c96b775eb0f489099f6169dd85b2683 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:31.937693 17790 raft_consensus.cc:2454] T c166820f9a314c0da2914de636252c0d P 3c96b775eb0f489099f6169dd85b2683 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22f4448136b4b06945ad3ffa8b0231a in term 1.
I20260620 19:10:31.937693 17510 raft_consensus.cc:2454] T c166820f9a314c0da2914de636252c0d P 35a74422fbe4496abc46cf71d22af858 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22f4448136b4b06945ad3ffa8b0231a in term 1.
I20260620 19:10:31.938048 17581 leader_election.cc:302] T c166820f9a314c0da2914de636252c0d P b22f4448136b4b06945ad3ffa8b0231a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 35a74422fbe4496abc46cf71d22af858, 3c96b775eb0f489099f6169dd85b2683, b22f4448136b4b06945ad3ffa8b0231a; no voters:
I20260620 19:10:31.938232 17874 raft_consensus.cc:2781] T c166820f9a314c0da2914de636252c0d P b22f4448136b4b06945ad3ffa8b0231a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:10:31.938331 17874 raft_consensus.cc:686] T c166820f9a314c0da2914de636252c0d P b22f4448136b4b06945ad3ffa8b0231a [term 1 LEADER]: Becoming Leader. State: Replica: b22f4448136b4b06945ad3ffa8b0231a, State: Running, Role: LEADER
I20260620 19:10:31.938473 17874 consensus_queue.cc:237] T c166820f9a314c0da2914de636252c0d P b22f4448136b4b06945ad3ffa8b0231a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35a74422fbe4496abc46cf71d22af858" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 45895 } } peers { permanent_uuid: "b22f4448136b4b06945ad3ffa8b0231a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 39779 } } peers { permanent_uuid: "3c96b775eb0f489099f6169dd85b2683" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 42389 } }
I20260620 19:10:31.940115 17351 catalog_manager.cc:5495] T c166820f9a314c0da2914de636252c0d P b22f4448136b4b06945ad3ffa8b0231a reported cstate change: term changed from 0 to 1, leader changed from <none> to b22f4448136b4b06945ad3ffa8b0231a (127.9.155.66). New cstate: current_term: 1 leader_uuid: "b22f4448136b4b06945ad3ffa8b0231a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35a74422fbe4496abc46cf71d22af858" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 45895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b22f4448136b4b06945ad3ffa8b0231a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 39779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c96b775eb0f489099f6169dd85b2683" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 42389 } health_report { overall_health: UNKNOWN } } }
I20260620 19:10:31.983817 17790 raft_consensus.cc:1261] T c166820f9a314c0da2914de636252c0d P 3c96b775eb0f489099f6169dd85b2683 [term 1 FOLLOWER]: Refusing update from remote peer b22f4448136b4b06945ad3ffa8b0231a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:10:31.984195 17510 raft_consensus.cc:1261] T c166820f9a314c0da2914de636252c0d P 35a74422fbe4496abc46cf71d22af858 [term 1 FOLLOWER]: Refusing update from remote peer b22f4448136b4b06945ad3ffa8b0231a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:10:31.984248 17872 consensus_queue.cc:1035] T c166820f9a314c0da2914de636252c0d P b22f4448136b4b06945ad3ffa8b0231a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c96b775eb0f489099f6169dd85b2683" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 42389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:10:31.984567 17874 consensus_queue.cc:1035] T c166820f9a314c0da2914de636252c0d P b22f4448136b4b06945ad3ffa8b0231a [LEADER]: Connected to new peer: Peer: permanent_uuid: "35a74422fbe4496abc46cf71d22af858" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 45895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:10:32.020131 17350 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52138:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:10:32.020316 17350 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52138:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:10:32.022198 17350 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 42920bad573b462593296d6a1276cb32: Sending DeleteTablet for 3 replicas of tablet c166820f9a314c0da2914de636252c0d
I20260620 19:10:32.022753 17490 tablet_service.cc:1514] Processing DeleteTablet for tablet c166820f9a314c0da2914de636252c0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:10:32 UTC) from {username='oryx', principal='oryx/127.9.155.126@KRBTEST.COM'} at 127.0.0.1:48450
I20260620 19:10:32.022784 17770 tablet_service.cc:1514] Processing DeleteTablet for tablet c166820f9a314c0da2914de636252c0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:10:32 UTC) from {username='oryx', principal='oryx/127.9.155.126@KRBTEST.COM'} at 127.0.0.1:57906
I20260620 19:10:32.022783 17630 tablet_service.cc:1514] Processing DeleteTablet for tablet c166820f9a314c0da2914de636252c0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:10:32 UTC) from {username='oryx', principal='oryx/127.9.155.126@KRBTEST.COM'} at 127.0.0.1:48394
I20260620 19:10:32.022957 17888 tablet_replica.cc:330] T c166820f9a314c0da2914de636252c0d P 35a74422fbe4496abc46cf71d22af858: stopping tablet replica
I20260620 19:10:32.022987 17887 tablet_replica.cc:330] T c166820f9a314c0da2914de636252c0d P b22f4448136b4b06945ad3ffa8b0231a: stopping tablet replica
I20260620 19:10:32.023087 17888 raft_consensus.cc:2229] T c166820f9a314c0da2914de636252c0d P 35a74422fbe4496abc46cf71d22af858 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:10:32.023141 17887 raft_consensus.cc:2229] T c166820f9a314c0da2914de636252c0d P b22f4448136b4b06945ad3ffa8b0231a [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:10:32.023204 17888 raft_consensus.cc:2258] T c166820f9a314c0da2914de636252c0d P 35a74422fbe4496abc46cf71d22af858 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:10:32.023300 17887 raft_consensus.cc:2258] T c166820f9a314c0da2914de636252c0d P b22f4448136b4b06945ad3ffa8b0231a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:10:32.023437 17889 tablet_replica.cc:330] T c166820f9a314c0da2914de636252c0d P 3c96b775eb0f489099f6169dd85b2683: stopping tablet replica
I20260620 19:10:32.023547 17889 raft_consensus.cc:2229] T c166820f9a314c0da2914de636252c0d P 3c96b775eb0f489099f6169dd85b2683 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:10:32.023554 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 17425
I20260620 19:10:32.023648 17889 raft_consensus.cc:2258] T c166820f9a314c0da2914de636252c0d P 3c96b775eb0f489099f6169dd85b2683 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:10:32.024986 17887 ts_tablet_manager.cc:1905] T c166820f9a314c0da2914de636252c0d P b22f4448136b4b06945ad3ffa8b0231a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:10:32.025527 17889 ts_tablet_manager.cc:1905] T c166820f9a314c0da2914de636252c0d P 3c96b775eb0f489099f6169dd85b2683: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:10:32.026944 17887 ts_tablet_manager.cc:1918] T c166820f9a314c0da2914de636252c0d P b22f4448136b4b06945ad3ffa8b0231a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 19:10:32.026996 17887 log.cc:1196] T c166820f9a314c0da2914de636252c0d P b22f4448136b4b06945ad3ffa8b0231a: Deleting WAL directory at /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/minicluster-data/ts-1/wal/wals/c166820f9a314c0da2914de636252c0d
I20260620 19:10:32.027166 17887 ts_tablet_manager.cc:1939] T c166820f9a314c0da2914de636252c0d P b22f4448136b4b06945ad3ffa8b0231a: Deleting consensus metadata
I20260620 19:10:32.027341 17889 ts_tablet_manager.cc:1918] T c166820f9a314c0da2914de636252c0d P 3c96b775eb0f489099f6169dd85b2683: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 19:10:32.027419 17889 log.cc:1196] T c166820f9a314c0da2914de636252c0d P 3c96b775eb0f489099f6169dd85b2683: Deleting WAL directory at /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982595231230-9837-0/minicluster-data/ts-2/wal/wals/c166820f9a314c0da2914de636252c0d
I20260620 19:10:32.027617 17889 ts_tablet_manager.cc:1939] T c166820f9a314c0da2914de636252c0d P 3c96b775eb0f489099f6169dd85b2683: Deleting consensus metadata
I20260620 19:10:32.027884 17340 catalog_manager.cc:4841] TS b22f4448136b4b06945ad3ffa8b0231a (127.9.155.66:39779): tablet c166820f9a314c0da2914de636252c0d (table test-table [id=51f5e82abe4242e795eab664be95bda6]) successfully deleted
I20260620 19:10:32.028267 17339 catalog_manager.cc:4841] TS 3c96b775eb0f489099f6169dd85b2683 (127.9.155.67:42389): tablet c166820f9a314c0da2914de636252c0d (table test-table [id=51f5e82abe4242e795eab664be95bda6]) successfully deleted
W20260620 19:10:32.031373 17337 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.155.65:45895 (error 108)
I20260620 19:10:32.032258 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 17565
I20260620 19:10:32.032418 17951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:32.032045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.65:45895 (local address 127.0.0.1:48462)
0620 19:10:32.032259 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:32.032319 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.65:45895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":57,"threads_started":1}
W20260620 19:10:32.033427 17337 catalog_manager.cc:4584] TS 35a74422fbe4496abc46cf71d22af858 (127.9.155.65:45895): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c166820f9a314c0da2914de636252c0d: Network error: Client connection negotiation failed: client connection to 127.9.155.65:45895: connect: Connection refused (error 111)
I20260620 19:10:32.038818 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 17705
I20260620 19:10:32.044412 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 17321
2026-06-20T19:10:32Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3321 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781982595231230-9837-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:10:32 dist-test-slave-n3fx krb5kdc[17954](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:10:32 dist-test-slave-n3fx krb5kdc[17954](info): set up 2 sockets
Jun 20 19:10:32 dist-test-slave-n3fx krb5kdc[17954](info): commencing operation
krb5kdc: starting...
W20260620 19:10:33.005142 9837 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781982595231230-9837-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781982595231230-9837-0/krb5kdc/test-user.keytab.
Jun 20 19:10:33 dist-test-slave-n3fx krb5kdc[17954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982633, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:10:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:10:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.155.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.155.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.155.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab.
Entry for principal kudu/127.9.155.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab.
I20260620 19:10:33.126590 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781982595231230-9837-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781982595231230-9837-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781982595231230-9837-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:33505
--webserver_interface=127.9.155.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab
--principal=kudu/127.9.155.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:44817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.155.126:33505
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:10:33.254920 17967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:33.255221 17967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:33.255262 17967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:33.259474 17967 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:10:33.259562 17967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:33.259580 17967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:33.259595 17967 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:10:33.259609 17967 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:10:33.265070 17967 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:44817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781982595231230-9837-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781982595231230-9837-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.155.126:33505
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:33505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.155.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17967
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781982595231230-9837-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:10:33.266126 17967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:33.267110 17967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:10:33.275597 17973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:33.275638 17967 server_base.cc:1034] running on GCE node
W20260620 19:10:33.275599 17972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:33.275597 17975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:33.276530 17967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:33.277693 17967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:33.278872 17967 hybrid_clock.cc:648] HybridClock initialized: now 1781982633278843 us; error 42 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-20T19:10:33Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1240 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:10:33 dist-test-slave-n3fx krb5kdc[17980](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:10:33 dist-test-slave-n3fx krb5kdc[17980](info): set up 2 sockets
Jun 20 19:10:33 dist-test-slave-n3fx krb5kdc[17980](info): commencing operation
krb5kdc: starting...
W20260620 19:10:34.240651 9837 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/krb5kdc/test-user.keytab.
Jun 20 19:10:34 dist-test-slave-n3fx krb5kdc[17980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:10:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:10:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.155.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.155.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.155.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab.
Entry for principal kudu/127.9.155.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab.
I20260620 19:10:34.358575 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:40955
--webserver_interface=127.9.155.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab
--principal=kudu/127.9.155.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:46433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.155.126:40955
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:10:34.468286 17993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:34.468585 17993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:34.468627 17993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:34.472827 17993 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:10:34.472910 17993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:34.472929 17993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:34.472945 17993 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:10:34.472960 17993 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:10:34.478440 17993 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:46433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.155.126:40955
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab
--principal=kudu/127.9.155.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:40955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.155.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17993
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:10:34.479545 17993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:34.480598 17993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:10:34.489166 18001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:34.489168 17999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:34.489166 17998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:34.489634 17993 server_base.cc:1034] running on GCE node
I20260620 19:10:34.490190 17993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:34.491323 17993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:34.492542 17993 hybrid_clock.cc:648] HybridClock initialized: now 1781982634492529 us; error 35 us; skew 500 ppm
Jun 20 19:10:34 dist-test-slave-n3fx krb5kdc[17980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982634, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:34.495592 17993 init.cc:377] Logged in from keytab as kudu/127.9.155.126@KRBTEST.COM (short username kudu)
I20260620 19:10:34.496621 17993 webserver.cc:460] Webserver started at http://127.9.155.126:34169/ using document root <none> and password file <none>
I20260620 19:10:34.497196 17993 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:34.497268 17993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:34.497473 17993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:10:34.499554 17993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/master-0/data/instance:
uuid: "b27186a078c64082a6c50c2ba7c6e132"
format_stamp: "Formatted at 2026-06-20 19:10:34 on dist-test-slave-n3fx"
server_key: "439135163bc77c98fac9f6a11f9b6a72"
server_key_iv: "5432863deb7ccb9a27c7f3311a979f61"
server_key_version: "encryptionkey@0"
I20260620 19:10:34.500142 17993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/master-0/wal/instance:
uuid: "b27186a078c64082a6c50c2ba7c6e132"
format_stamp: "Formatted at 2026-06-20 19:10:34 on dist-test-slave-n3fx"
server_key: "439135163bc77c98fac9f6a11f9b6a72"
server_key_iv: "5432863deb7ccb9a27c7f3311a979f61"
server_key_version: "encryptionkey@0"
I20260620 19:10:34.504143 17993 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 19:10:34.506832 18008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:34.508249 17993 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:10:34.508399 17993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/master-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/master-0/wal
uuid: "b27186a078c64082a6c50c2ba7c6e132"
format_stamp: "Formatted at 2026-06-20 19:10:34 on dist-test-slave-n3fx"
server_key: "439135163bc77c98fac9f6a11f9b6a72"
server_key_iv: "5432863deb7ccb9a27c7f3311a979f61"
server_key_version: "encryptionkey@0"
I20260620 19:10:34.508564 17993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:34.525305 17993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:34.528903 17993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:34.529094 17993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:34.536371 17993 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.126:40955
I20260620 19:10:34.536445 18060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.126:40955 every 8 connection(s)
I20260620 19:10:34.537467 17993 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
I20260620 19:10:34.540352 18061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:34.545049 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 17993
I20260620 19:10:34.545163 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/master-0/wal/instance
I20260620 19:10:34.545409 9837 external_mini_cluster.cc:1428] Setting key 69bb1f3c11ed56b2d0e3dc8b35b14058
I20260620 19:10:34.546217 18061 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b27186a078c64082a6c50c2ba7c6e132: Bootstrap starting.
I20260620 19:10:34.548978 18061 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b27186a078c64082a6c50c2ba7c6e132: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:34.549836 18061 log.cc:824] T 00000000000000000000000000000000 P b27186a078c64082a6c50c2ba7c6e132: Log is configured to *not* fsync() on all Append() calls
I20260620 19:10:34.551996 18061 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b27186a078c64082a6c50c2ba7c6e132: No bootstrap required, opened a new log
Jun 20 19:10:34 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:10:34.555733 18061 raft_consensus.cc:348] T 00000000000000000000000000000000 P b27186a078c64082a6c50c2ba7c6e132 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b27186a078c64082a6c50c2ba7c6e132" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 40955 } }
I20260620 19:10:34.556012 18061 raft_consensus.cc:374] T 00000000000000000000000000000000 P b27186a078c64082a6c50c2ba7c6e132 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:34.556082 18061 raft_consensus.cc:729] T 00000000000000000000000000000000 P b27186a078c64082a6c50c2ba7c6e132 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b27186a078c64082a6c50c2ba7c6e132, State: Initialized, Role: FOLLOWER
I20260620 19:10:34.556565 18061 consensus_queue.cc:260] T 00000000000000000000000000000000 P b27186a078c64082a6c50c2ba7c6e132 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b27186a078c64082a6c50c2ba7c6e132" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 40955 } }
I20260620 19:10:34.556689 18061 raft_consensus.cc:388] T 00000000000000000000000000000000 P b27186a078c64082a6c50c2ba7c6e132 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:10:34.556747 18061 raft_consensus.cc:482] T 00000000000000000000000000000000 P b27186a078c64082a6c50c2ba7c6e132 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:10:34.556839 18061 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b27186a078c64082a6c50c2ba7c6e132 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:34.557708 18061 raft_consensus.cc:504] T 00000000000000000000000000000000 P b27186a078c64082a6c50c2ba7c6e132 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b27186a078c64082a6c50c2ba7c6e132" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 40955 } }
I20260620 19:10:34.558022 18061 leader_election.cc:302] T 00000000000000000000000000000000 P b27186a078c64082a6c50c2ba7c6e132 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b27186a078c64082a6c50c2ba7c6e132; no voters:
I20260620 19:10:34.558305 18061 leader_election.cc:288] T 00000000000000000000000000000000 P b27186a078c64082a6c50c2ba7c6e132 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:10:34.558393 18066 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b27186a078c64082a6c50c2ba7c6e132 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:10:34.558555 18066 raft_consensus.cc:686] T 00000000000000000000000000000000 P b27186a078c64082a6c50c2ba7c6e132 [term 1 LEADER]: Becoming Leader. State: Replica: b27186a078c64082a6c50c2ba7c6e132, State: Running, Role: LEADER
I20260620 19:10:34.558928 18066 consensus_queue.cc:237] T 00000000000000000000000000000000 P b27186a078c64082a6c50c2ba7c6e132 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b27186a078c64082a6c50c2ba7c6e132" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 40955 } }
I20260620 19:10:34.559638 18061 sys_catalog.cc:564] T 00000000000000000000000000000000 P b27186a078c64082a6c50c2ba7c6e132 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:10:34.560048 18064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:34.546929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32928 (local address 127.9.155.126:40955)
0620 19:10:34.547379 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:34.547400 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:34.547447 (+ 47us) server_negotiation.cc:407] Connection header received
0620 19:10:34.548327 (+ 880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:34.548351 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:34.548767 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:34.549143 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:34.549831 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:34.551098 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:34.552007 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:34.552699 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:34.552861 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:34.555310 (+ 2449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:34.555316 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:34.555333 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:34.555364 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:34.557152 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:34.557551 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:34.557557 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:34.557564 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:34.557680 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:34.557953 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:34.557956 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:34.557958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:34.558438 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:34.558666 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:34.558958 (+ 292us) server_negotiation.cc:299] Negotiation successful
0620 19:10:34.559257 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":118,"threads_started":1}
I20260620 19:10:34.560573 18068 sys_catalog.cc:455] T 00000000000000000000000000000000 P b27186a078c64082a6c50c2ba7c6e132 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b27186a078c64082a6c50c2ba7c6e132" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b27186a078c64082a6c50c2ba7c6e132" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 40955 } } }
I20260620 19:10:34.560701 18068 sys_catalog.cc:458] T 00000000000000000000000000000000 P b27186a078c64082a6c50c2ba7c6e132 [sys.catalog]: This master's current role is: LEADER
I20260620 19:10:34.561028 18067 sys_catalog.cc:455] T 00000000000000000000000000000000 P b27186a078c64082a6c50c2ba7c6e132 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b27186a078c64082a6c50c2ba7c6e132. Latest consensus state: current_term: 1 leader_uuid: "b27186a078c64082a6c50c2ba7c6e132" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b27186a078c64082a6c50c2ba7c6e132" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 40955 } } }
I20260620 19:10:34.561089 18075 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:10:34.561105 18067 sys_catalog.cc:458] T 00000000000000000000000000000000 P b27186a078c64082a6c50c2ba7c6e132 [sys.catalog]: This master's current role is: LEADER
I20260620 19:10:34.564189 18075 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:10:34.570098 18075 catalog_manager.cc:1347] Generated new cluster ID: fa21c1eaad6a4dbba0e75ef5042ae6b7
I20260620 19:10:34.570176 18075 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:10:34.596062 18075 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:10:34.597195 18075 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:10:34.611169 18075 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b27186a078c64082a6c50c2ba7c6e132: Generated new TSK 0
I20260620 19:10:34.612095 18075 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.155.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.155.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.155.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.65.keytab.
Entry for principal kudu/127.9.155.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.65.keytab.
I20260620 19:10:34.641289 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.65:0
--local_ip_for_outbound_sockets=127.9.155.65
--webserver_interface=127.9.155.65
--webserver_port=0
--tserver_master_addrs=127.9.155.126:40955
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.65.keytab
--principal=kudu/127.9.155.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:46433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:10:34.753610 18087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:34.753903 18087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:34.753947 18087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:34.758013 18087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:34.758082 18087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:34.758143 18087 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.65
I20260620 19:10:34.763720 18087 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:46433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.65.keytab
--principal=kudu/127.9.155.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.155.65
--webserver_port=0
--tserver_master_addrs=127.9.155.126:40955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.65
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:10:34.764838 18087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:34.765810 18087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:10:34.775611 18095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:34.775630 18093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:34.775664 18087 server_base.cc:1034] running on GCE node
W20260620 19:10:34.775614 18092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:34.776283 18087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:34.776955 18087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:34.778124 18087 hybrid_clock.cc:648] HybridClock initialized: now 1781982634778100 us; error 35 us; skew 500 ppm
Jun 20 19:10:34 dist-test-slave-n3fx krb5kdc[17980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982634, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:34.781097 18087 init.cc:377] Logged in from keytab as kudu/127.9.155.65@KRBTEST.COM (short username kudu)
I20260620 19:10:34.782044 18087 webserver.cc:460] Webserver started at http://127.9.155.65:43177/ using document root <none> and password file <none>
I20260620 19:10:34.782588 18087 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:34.782649 18087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:34.782848 18087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:10:34.785169 18087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-0/data/instance:
uuid: "88eb837a8d6d4d24aa96c1e1e78d0224"
format_stamp: "Formatted at 2026-06-20 19:10:34 on dist-test-slave-n3fx"
server_key: "584fd60d5e1109e18fbd0a0a55ac29c7"
server_key_iv: "82b360094b22a3161253326526b89072"
server_key_version: "encryptionkey@0"
I20260620 19:10:34.785761 18087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-0/wal/instance:
uuid: "88eb837a8d6d4d24aa96c1e1e78d0224"
format_stamp: "Formatted at 2026-06-20 19:10:34 on dist-test-slave-n3fx"
server_key: "584fd60d5e1109e18fbd0a0a55ac29c7"
server_key_iv: "82b360094b22a3161253326526b89072"
server_key_version: "encryptionkey@0"
I20260620 19:10:34.789798 18087 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 19:10:34.792634 18102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:34.794013 18087 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 19:10:34.794147 18087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-0/wal
uuid: "88eb837a8d6d4d24aa96c1e1e78d0224"
format_stamp: "Formatted at 2026-06-20 19:10:34 on dist-test-slave-n3fx"
server_key: "584fd60d5e1109e18fbd0a0a55ac29c7"
server_key_iv: "82b360094b22a3161253326526b89072"
server_key_version: "encryptionkey@0"
I20260620 19:10:34.794262 18087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:34.818941 18087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:34.822126 18087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:34.822309 18087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:34.822913 18087 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:10:34.823774 18087 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:10:34.823824 18087 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:34.823877 18087 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:10:34.823907 18087 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:34.832615 18087 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.65:41923
I20260620 19:10:34.832643 18215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.65:41923 every 8 connection(s)
I20260620 19:10:34.833530 18087 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
I20260620 19:10:34.837949 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 18087
I20260620 19:10:34.838079 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-0/wal/instance
I20260620 19:10:34.838390 9837 external_mini_cluster.cc:1428] Setting key 7265fc27743b23cba59720207f8603ed
Jun 20 19:10:34 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982634, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.65@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.155.66@KRBTEST.COM; defaulting to no policy
I20260620 19:10:34.847297 18064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:34.835390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.65:55879 (local address 127.9.155.126:40955)
0620 19:10:34.835600 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:34.835604 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:34.836576 (+ 972us) server_negotiation.cc:407] Connection header received
0620 19:10:34.837447 (+ 871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:34.837450 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:34.837497 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:34.837572 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:34.839037 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:34.839568 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:34.840758 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:34.840984 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:34.841074 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:34.843756 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:34.843760 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:34.843762 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:34.843787 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:34.845108 (+ 1321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:34.845715 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:34.845717 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:34.845719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:34.845790 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:34.846133 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:34.846135 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:34.846136 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:34.846317 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:34.846386 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:34.847060 (+ 674us) server_negotiation.cc:299] Negotiation successful
0620 19:10:34.847175 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
I20260620 19:10:34.848233 18218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:34.835746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40955 (local address 127.9.155.65:55879)
0620 19:10:34.836431 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:34.836472 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:34.837239 (+ 767us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:34.837743 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:34.837752 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:34.838261 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:34.838889 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:34.838907 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:34.839693 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:34.839696 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:34.840637 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:34.840647 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:34.841124 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:34.841128 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:34.841287 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:34.841831 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:34.841847 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:34.843561 (+ 1714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:34.845226 (+ 1665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:34.845234 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:34.845253 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:34.845619 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:34.845869 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:34.845872 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:34.845875 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:34.846051 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:34.846424 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:34.846432 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:34.846779 (+ 347us) client_negotiation.cc:769] Sending connection context
0620 19:10:34.846989 (+ 210us) client_negotiation.cc:240] Negotiation successful
0620 19:10:34.847344 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":126,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.155.66@KRBTEST.COM" created.
I20260620 19:10:34.849509 18216 heartbeater.cc:347] Connected to a master server at 127.9.155.126:40955
I20260620 19:10:34.849730 18216 heartbeater.cc:464] Registering TS with master...
I20260620 19:10:34.850224 18216 heartbeater.cc:511] Master 127.9.155.126:40955 requested a full tablet report, sending...
I20260620 19:10:34.851696 18025 ts_manager.cc:194] Registered new tserver with Master: 88eb837a8d6d4d24aa96c1e1e78d0224 (127.9.155.65:41923)
I20260620 19:10:34.852854 18025 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.155.65@KRBTEST.COM'} at 127.9.155.65:55879
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.155.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.66.keytab.
Entry for principal kudu/127.9.155.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.66.keytab.
I20260620 19:10:34.867079 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.66:0
--local_ip_for_outbound_sockets=127.9.155.66
--webserver_interface=127.9.155.66
--webserver_port=0
--tserver_master_addrs=127.9.155.126:40955
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.66.keytab
--principal=kudu/127.9.155.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:46433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:10:34.979578 18221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:34.979882 18221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:34.979926 18221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:34.984754 18221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:34.984839 18221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:34.984907 18221 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.66
I20260620 19:10:34.990594 18221 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:46433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.66.keytab
--principal=kudu/127.9.155.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.155.66
--webserver_port=0
--tserver_master_addrs=127.9.155.126:40955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18221
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.66
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:10:34.991691 18221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:34.992795 18221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:10:35.002713 18226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:35.002810 18221 server_base.cc:1034] running on GCE node
W20260620 19:10:35.002705 18229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:35.002979 18227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:35.003376 18221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:35.004071 18221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:35.005193 18221 hybrid_clock.cc:648] HybridClock initialized: now 1781982635005128 us; error 82 us; skew 500 ppm
Jun 20 19:10:35 dist-test-slave-n3fx krb5kdc[17980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982635, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:35.011734 18221 init.cc:377] Logged in from keytab as kudu/127.9.155.66@KRBTEST.COM (short username kudu)
I20260620 19:10:35.012902 18221 webserver.cc:460] Webserver started at http://127.9.155.66:33141/ using document root <none> and password file <none>
I20260620 19:10:35.013494 18221 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:35.013561 18221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:35.013728 18221 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:10:35.015749 18221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-1/data/instance:
uuid: "140a6670e6f74a5d9ed996a741cc352b"
format_stamp: "Formatted at 2026-06-20 19:10:35 on dist-test-slave-n3fx"
server_key: "4f1a8bc1996986bc800d5a1b763ce618"
server_key_iv: "f8188a6bee05d3320a172b547a906fa2"
server_key_version: "encryptionkey@0"
I20260620 19:10:35.016324 18221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-1/wal/instance:
uuid: "140a6670e6f74a5d9ed996a741cc352b"
format_stamp: "Formatted at 2026-06-20 19:10:35 on dist-test-slave-n3fx"
server_key: "4f1a8bc1996986bc800d5a1b763ce618"
server_key_iv: "f8188a6bee05d3320a172b547a906fa2"
server_key_version: "encryptionkey@0"
I20260620 19:10:35.020136 18221 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 19:10:35.022699 18236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:35.023976 18221 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 19:10:35.024083 18221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-1/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-1/wal
uuid: "140a6670e6f74a5d9ed996a741cc352b"
format_stamp: "Formatted at 2026-06-20 19:10:35 on dist-test-slave-n3fx"
server_key: "4f1a8bc1996986bc800d5a1b763ce618"
server_key_iv: "f8188a6bee05d3320a172b547a906fa2"
server_key_version: "encryptionkey@0"
I20260620 19:10:35.024197 18221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:35.041005 18221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:35.044286 18221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:35.044476 18221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:35.045101 18221 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:10:35.045931 18221 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:10:35.045977 18221 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:35.046021 18221 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:10:35.046051 18221 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:35.054656 18221 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.66:35221
I20260620 19:10:35.054689 18349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.66:35221 every 8 connection(s)
I20260620 19:10:35.055568 18221 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
I20260620 19:10:35.063843 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 18221
I20260620 19:10:35.063963 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-1/wal/instance
I20260620 19:10:35.064263 9837 external_mini_cluster.cc:1428] Setting key 6530a1ebb343ac96aa2770315c16cc32
Jun 20 19:10:35 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982635, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.66@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:10:35.070300 18064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:35.057321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.66:38175 (local address 127.9.155.126:40955)
0620 19:10:35.057485 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:35.057490 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:35.058583 (+ 1093us) server_negotiation.cc:407] Connection header received
0620 19:10:35.059457 (+ 874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:35.059460 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:35.059514 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:35.059595 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:35.061081 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:35.061728 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:35.063138 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:35.063365 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:35.063464 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:35.066240 (+ 2776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:35.066244 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:35.066246 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:35.066274 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:35.067826 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:35.068607 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:35.068610 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:35.068612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:35.068704 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:35.069079 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:35.069082 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:35.069083 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:35.069274 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:35.069349 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:35.070052 (+ 703us) server_negotiation.cc:299] Negotiation successful
0620 19:10:35.070166 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 19:10:35.071482 18352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:35.057759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40955 (local address 127.9.155.66:38175)
0620 19:10:35.058420 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:35.058470 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:35.059263 (+ 793us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:35.059799 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:35.059806 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:35.060299 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:35.060885 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:35.060911 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:35.061871 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:35.061875 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:35.062981 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:35.062992 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:35.063537 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:35.063540 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:35.063727 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:35.064396 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:35.064415 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:35.066048 (+ 1633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:35.067959 (+ 1911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:35.067966 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:35.067987 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:35.068493 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:35.068795 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:35.068797 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:35.068800 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:35.068992 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:35.069367 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:35.069374 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:35.069722 (+ 348us) client_negotiation.cc:769] Sending connection context
0620 19:10:35.069957 (+ 235us) client_negotiation.cc:240] Negotiation successful
0620 19:10:35.070349 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":118,"threads_started":1}
I20260620 19:10:35.072993 18350 heartbeater.cc:347] Connected to a master server at 127.9.155.126:40955
I20260620 19:10:35.073264 18350 heartbeater.cc:464] Registering TS with master...
I20260620 19:10:35.073889 18350 heartbeater.cc:511] Master 127.9.155.126:40955 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.9.155.67@KRBTEST.COM; defaulting to no policy
I20260620 19:10:35.075109 18025 ts_manager.cc:194] Registered new tserver with Master: 140a6670e6f74a5d9ed996a741cc352b (127.9.155.66:35221)
I20260620 19:10:35.075765 18025 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.155.66@KRBTEST.COM'} at 127.9.155.66:38175
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.155.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.155.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.67.keytab.
Entry for principal kudu/127.9.155.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.67.keytab.
I20260620 19:10:35.091637 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.67:0
--local_ip_for_outbound_sockets=127.9.155.67
--webserver_interface=127.9.155.67
--webserver_port=0
--tserver_master_addrs=127.9.155.126:40955
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.67.keytab
--principal=kudu/127.9.155.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:46433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:10:35.204306 18355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:35.204612 18355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:35.204674 18355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:35.209108 18355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:35.209190 18355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:35.209285 18355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.67
I20260620 19:10:35.215027 18355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:46433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.67.keytab
--principal=kudu/127.9.155.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.155.67
--webserver_port=0
--tserver_master_addrs=127.9.155.126:40955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18355
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.67
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:10:35.216209 18355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:35.217280 18355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:10:35.227097 18363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:35.227108 18360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:35.227108 18361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:35.227504 18355 server_base.cc:1034] running on GCE node
I20260620 19:10:35.228047 18355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:35.228826 18355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:35.229997 18355 hybrid_clock.cc:648] HybridClock initialized: now 1781982635229978 us; error 35 us; skew 500 ppm
Jun 20 19:10:35 dist-test-slave-n3fx krb5kdc[17980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982635, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:35.232918 18355 init.cc:377] Logged in from keytab as kudu/127.9.155.67@KRBTEST.COM (short username kudu)
I20260620 19:10:35.233911 18355 webserver.cc:460] Webserver started at http://127.9.155.67:39813/ using document root <none> and password file <none>
I20260620 19:10:35.234464 18355 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:35.234532 18355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:35.234741 18355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:10:35.236877 18355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-2/data/instance:
uuid: "e7bd50dbb9794f36a99b908294d57efd"
format_stamp: "Formatted at 2026-06-20 19:10:35 on dist-test-slave-n3fx"
server_key: "eb9dde68e6851216aa264a82734a494a"
server_key_iv: "4added1cc4efa6da9ae0e8937be653ce"
server_key_version: "encryptionkey@0"
I20260620 19:10:35.237501 18355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-2/wal/instance:
uuid: "e7bd50dbb9794f36a99b908294d57efd"
format_stamp: "Formatted at 2026-06-20 19:10:35 on dist-test-slave-n3fx"
server_key: "eb9dde68e6851216aa264a82734a494a"
server_key_iv: "4added1cc4efa6da9ae0e8937be653ce"
server_key_version: "encryptionkey@0"
I20260620 19:10:35.241410 18355 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 19:10:35.244006 18370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:35.245302 18355 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:10:35.245483 18355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-2/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-2/wal
uuid: "e7bd50dbb9794f36a99b908294d57efd"
format_stamp: "Formatted at 2026-06-20 19:10:35 on dist-test-slave-n3fx"
server_key: "eb9dde68e6851216aa264a82734a494a"
server_key_iv: "4added1cc4efa6da9ae0e8937be653ce"
server_key_version: "encryptionkey@0"
I20260620 19:10:35.245604 18355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:35.261693 18355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:35.264923 18355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:35.265117 18355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:35.265715 18355 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:10:35.266785 18355 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:10:35.266845 18355 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:35.266906 18355 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:10:35.266933 18355 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:35.275492 18355 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.67:33055
I20260620 19:10:35.275518 18483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.67:33055 every 8 connection(s)
I20260620 19:10:35.276424 18355 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
I20260620 19:10:35.278205 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 18355
I20260620 19:10:35.278332 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-2/wal/instance
I20260620 19:10:35.278635 9837 external_mini_cluster.cc:1428] Setting key c1b7f442ccaf383c800c60a859606360
Jun 20 19:10:35 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982635, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.67@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:10:35.290474 18064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:35.278377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.67:55789 (local address 127.9.155.126:40955)
0620 19:10:35.278539 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:35.278543 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:35.279592 (+ 1049us) server_negotiation.cc:407] Connection header received
0620 19:10:35.281076 (+ 1484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:35.281078 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:35.281126 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:35.281198 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:35.282538 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:35.283070 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:35.284263 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:35.284521 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:35.284613 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:35.287122 (+ 2509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:35.287125 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:35.287127 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:35.287152 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:35.288465 (+ 1313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:35.288979 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:35.288982 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:35.288983 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:35.289069 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:35.289349 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:35.289351 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:35.289352 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:35.289542 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:35.289617 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:35.290193 (+ 576us) server_negotiation.cc:299] Negotiation successful
0620 19:10:35.290296 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 19:10:35.291390 18486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:35.278779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40955 (local address 127.9.155.67:55789)
0620 19:10:35.279407 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:35.279472 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:35.280854 (+ 1382us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:35.281407 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:35.281415 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:35.281817 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:35.282372 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:35.282387 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:35.283187 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:35.283190 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:35.284102 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:35.284112 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:35.284664 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:35.284667 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:35.284819 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:35.285400 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:35.285418 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:35.286940 (+ 1522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:35.288567 (+ 1627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:35.288573 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:35.288587 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:35.288864 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:35.289148 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:35.289151 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:35.289152 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:35.289283 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:35.289652 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:35.289659 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:35.289984 (+ 325us) client_negotiation.cc:769] Sending connection context
0620 19:10:35.290161 (+ 177us) client_negotiation.cc:240] Negotiation successful
0620 19:10:35.290452 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":148,"threads_started":1}
I20260620 19:10:35.292613 18484 heartbeater.cc:347] Connected to a master server at 127.9.155.126:40955
I20260620 19:10:35.292881 18484 heartbeater.cc:464] Registering TS with master...
I20260620 19:10:35.293381 18484 heartbeater.cc:511] Master 127.9.155.126:40955 requested a full tablet report, sending...
I20260620 19:10:35.294389 18024 ts_manager.cc:194] Registered new tserver with Master: e7bd50dbb9794f36a99b908294d57efd (127.9.155.67:33055)
I20260620 19:10:35.294984 18024 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.155.67@KRBTEST.COM'} at 127.9.155.67:55789
I20260620 19:10:35.304998 9837 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.9.155.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.155.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.155.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/krb5kdc/oryx_127.9.155.65.keytab.
Entry for principal oryx/127.9.155.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/krb5kdc/oryx_127.9.155.65.keytab.
WARNING: no policy specified for oryx/127.9.155.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.155.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.155.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/krb5kdc/oryx_127.9.155.66.keytab.
Entry for principal oryx/127.9.155.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/krb5kdc/oryx_127.9.155.66.keytab.
WARNING: no policy specified for oryx/127.9.155.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.155.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.155.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/krb5kdc/oryx_127.9.155.67.keytab.
Entry for principal oryx/127.9.155.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/krb5kdc/oryx_127.9.155.67.keytab.
I20260620 19:10:35.378741 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 18087
I20260620 19:10:35.383744 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 18221
I20260620 19:10:35.388445 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 18355
I20260620 19:10:35.393227 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 17993
I20260620 19:10:35.398865 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:40955
--webserver_interface=127.9.155.126
--webserver_port=34169
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab
--principal=kudu/127.9.155.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:46433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.155.126:40955
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:10:35.509840 18493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:35.510125 18493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:35.510205 18493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:35.514318 18493 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:10:35.514390 18493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:35.514418 18493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:35.514536 18493 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:10:35.514560 18493 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:10:35.520041 18493 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:46433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.155.126:40955
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab
--principal=kudu/127.9.155.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:40955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.155.126
--webserver_port=34169
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18493
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:10:35.521185 18493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:35.522210 18493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:10:35.530910 18498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:35.530915 18499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:35.531023 18493 server_base.cc:1034] running on GCE node
W20260620 19:10:35.530920 18501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:35.531865 18493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:35.533087 18493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:35.534281 18493 hybrid_clock.cc:648] HybridClock initialized: now 1781982635534258 us; error 39 us; skew 500 ppm
Jun 20 19:10:35 dist-test-slave-n3fx krb5kdc[17980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982635, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:35.537380 18493 init.cc:377] Logged in from keytab as kudu/127.9.155.126@KRBTEST.COM (short username kudu)
I20260620 19:10:35.538405 18493 webserver.cc:460] Webserver started at http://127.9.155.126:34169/ using document root <none> and password file <none>
I20260620 19:10:35.538976 18493 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:35.539064 18493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:35.543138 18493 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 19:10:35.545394 18508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:35.546885 18493 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 19:10:35.547029 18493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/master-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/master-0/wal
uuid: "b27186a078c64082a6c50c2ba7c6e132"
format_stamp: "Formatted at 2026-06-20 19:10:34 on dist-test-slave-n3fx"
server_key: "439135163bc77c98fac9f6a11f9b6a72"
server_key_iv: "5432863deb7ccb9a27c7f3311a979f61"
server_key_version: "encryptionkey@0"
I20260620 19:10:35.547623 18493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:35.560165 18493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:35.563477 18493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:35.563660 18493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:35.570734 18493 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.126:40955
I20260620 19:10:35.570772 18560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.126:40955 every 8 connection(s)
I20260620 19:10:35.571827 18493 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
I20260620 19:10:35.574857 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 18493
I20260620 19:10:35.575276 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.65:41923
--local_ip_for_outbound_sockets=127.9.155.65
--tserver_master_addrs=127.9.155.126:40955
--webserver_port=43177
--webserver_interface=127.9.155.65
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.65.keytab
--principal=kudu/127.9.155.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:46433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.9.155.65
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/krb5kdc/oryx_127.9.155.65.keytab with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:10:35.576473 18561 sys_catalog.cc:263] Verifying existing consensus state
I20260620 19:10:35.578213 18561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b27186a078c64082a6c50c2ba7c6e132: Bootstrap starting.
I20260620 19:10:35.594671 18561 log.cc:824] T 00000000000000000000000000000000 P b27186a078c64082a6c50c2ba7c6e132: Log is configured to *not* fsync() on all Append() calls
I20260620 19:10:35.601523 18561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b27186a078c64082a6c50c2ba7c6e132: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:10:35.601874 18561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b27186a078c64082a6c50c2ba7c6e132: Bootstrap complete.
I20260620 19:10:35.605976 18561 raft_consensus.cc:348] T 00000000000000000000000000000000 P b27186a078c64082a6c50c2ba7c6e132 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b27186a078c64082a6c50c2ba7c6e132" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 40955 } }
I20260620 19:10:35.606514 18561 raft_consensus.cc:729] T 00000000000000000000000000000000 P b27186a078c64082a6c50c2ba7c6e132 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b27186a078c64082a6c50c2ba7c6e132, State: Initialized, Role: FOLLOWER
I20260620 19:10:35.607024 18561 consensus_queue.cc:260] T 00000000000000000000000000000000 P b27186a078c64082a6c50c2ba7c6e132 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b27186a078c64082a6c50c2ba7c6e132" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 40955 } }
I20260620 19:10:35.607153 18561 raft_consensus.cc:388] T 00000000000000000000000000000000 P b27186a078c64082a6c50c2ba7c6e132 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:10:35.607223 18561 raft_consensus.cc:482] T 00000000000000000000000000000000 P b27186a078c64082a6c50c2ba7c6e132 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:10:35.607314 18561 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b27186a078c64082a6c50c2ba7c6e132 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:10:35.609390 18561 raft_consensus.cc:504] T 00000000000000000000000000000000 P b27186a078c64082a6c50c2ba7c6e132 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b27186a078c64082a6c50c2ba7c6e132" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 40955 } }
I20260620 19:10:35.609716 18561 leader_election.cc:302] T 00000000000000000000000000000000 P b27186a078c64082a6c50c2ba7c6e132 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b27186a078c64082a6c50c2ba7c6e132; no voters:
I20260620 19:10:35.610036 18561 leader_election.cc:288] T 00000000000000000000000000000000 P b27186a078c64082a6c50c2ba7c6e132 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 19:10:35.610183 18565 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b27186a078c64082a6c50c2ba7c6e132 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:10:35.610383 18565 raft_consensus.cc:686] T 00000000000000000000000000000000 P b27186a078c64082a6c50c2ba7c6e132 [term 2 LEADER]: Becoming Leader. State: Replica: b27186a078c64082a6c50c2ba7c6e132, State: Running, Role: LEADER
I20260620 19:10:35.610754 18565 consensus_queue.cc:237] T 00000000000000000000000000000000 P b27186a078c64082a6c50c2ba7c6e132 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b27186a078c64082a6c50c2ba7c6e132" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 40955 } }
I20260620 19:10:35.611353 18561 sys_catalog.cc:564] T 00000000000000000000000000000000 P b27186a078c64082a6c50c2ba7c6e132 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:10:35.612202 18567 sys_catalog.cc:455] T 00000000000000000000000000000000 P b27186a078c64082a6c50c2ba7c6e132 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b27186a078c64082a6c50c2ba7c6e132. Latest consensus state: current_term: 2 leader_uuid: "b27186a078c64082a6c50c2ba7c6e132" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b27186a078c64082a6c50c2ba7c6e132" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 40955 } } }
I20260620 19:10:35.612367 18567 sys_catalog.cc:458] T 00000000000000000000000000000000 P b27186a078c64082a6c50c2ba7c6e132 [sys.catalog]: This master's current role is: LEADER
I20260620 19:10:35.612684 18566 sys_catalog.cc:455] T 00000000000000000000000000000000 P b27186a078c64082a6c50c2ba7c6e132 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b27186a078c64082a6c50c2ba7c6e132" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b27186a078c64082a6c50c2ba7c6e132" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 40955 } } }
I20260620 19:10:35.612761 18566 sys_catalog.cc:458] T 00000000000000000000000000000000 P b27186a078c64082a6c50c2ba7c6e132 [sys.catalog]: This master's current role is: LEADER
I20260620 19:10:35.612788 18574 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:10:35.616073 18574 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:10:35.616890 18574 catalog_manager.cc:1259] Loaded cluster ID: fa21c1eaad6a4dbba0e75ef5042ae6b7
I20260620 19:10:35.616953 18574 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:10:35.618722 18574 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:10:35.619470 18574 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b27186a078c64082a6c50c2ba7c6e132: Loaded TSK: 0
I20260620 19:10:35.620661 18574 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 19:10:35.699560 18563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:35.699855 18563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:35.699927 18563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:35.704080 18563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:35.704157 18563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:35.704258 18563 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.65
I20260620 19:10:35.709836 18563 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:46433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/krb5kdc/oryx_127.9.155.65.keytab
--principal=oryx/127.9.155.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.65:41923
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.155.65
--webserver_port=43177
--tserver_master_addrs=127.9.155.126:40955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18563
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.65
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:10:35.710875 18563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:35.712010 18563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:10:35.722412 18590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:35.722508 18588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:35.722512 18587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:35.722851 18563 server_base.cc:1034] running on GCE node
I20260620 19:10:35.723276 18563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:35.724036 18563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:35.725246 18563 hybrid_clock.cc:648] HybridClock initialized: now 1781982635725241 us; error 50 us; skew 500 ppm
Jun 20 19:10:35 dist-test-slave-n3fx krb5kdc[17980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982635, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.155.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:35.728276 18563 init.cc:377] Logged in from keytab as oryx/127.9.155.65@KRBTEST.COM (short username oryx)
I20260620 19:10:35.729302 18563 webserver.cc:460] Webserver started at http://127.9.155.65:43177/ using document root <none> and password file <none>
I20260620 19:10:35.729871 18563 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:35.729949 18563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:35.734026 18563 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 19:10:35.736244 18597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:35.737560 18563 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 19:10:35.737664 18563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-0/wal
uuid: "88eb837a8d6d4d24aa96c1e1e78d0224"
format_stamp: "Formatted at 2026-06-20 19:10:34 on dist-test-slave-n3fx"
server_key: "584fd60d5e1109e18fbd0a0a55ac29c7"
server_key_iv: "82b360094b22a3161253326526b89072"
server_key_version: "encryptionkey@0"
I20260620 19:10:35.738238 18563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:35.749079 18563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:35.752207 18563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:35.752411 18563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:35.753003 18563 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:10:35.753788 18563 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:10:35.753832 18563 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:35.753873 18563 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:10:35.753901 18563 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:35.764161 18563 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.65:41923
I20260620 19:10:35.764186 18710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.65:41923 every 8 connection(s)
I20260620 19:10:35.765244 18563 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
I20260620 19:10:35.771746 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 18563
I20260620 19:10:35.772259 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.66:35221
--local_ip_for_outbound_sockets=127.9.155.66
--tserver_master_addrs=127.9.155.126:40955
--webserver_port=33141
--webserver_interface=127.9.155.66
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.66.keytab
--principal=kudu/127.9.155.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:46433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.9.155.66
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/krb5kdc/oryx_127.9.155.66.keytab with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 19:10:35 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.65@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:10:35 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.65@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:10:35.778303 18714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:35.767424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40955 (local address 127.9.155.65:45061)
0620 19:10:35.768109 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:35.768160 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:35.768979 (+ 819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:35.770018 (+ 1039us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:35.770025 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:35.770395 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:35.770844 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:35.770858 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:35.772459 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:35.772463 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:35.773318 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:35.773327 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:35.774082 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:35.774087 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:35.774266 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:35.774827 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:35.774844 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:35.777483 (+ 2639us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.155.126:40955: Server oryx/127.9.155.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":113,"threads_started":1}
W20260620 19:10:35.778771 18713 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:10:35.767280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.65:45061 (local address 127.9.155.126:40955)
0620 19:10:35.767701 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:35.767717 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:35.768304 (+ 587us) server_negotiation.cc:407] Connection header received
0620 19:10:35.769344 (+ 1040us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:35.769355 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:35.769653 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:35.769918 (+ 265us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:35.771001 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:35.772274 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:35.773436 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:35.773922 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:35.774111 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:35.777820 (+ 3709us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.155.65:45061: BlockingRecv error: recv got EOF from 127.9.155.65:45061 (error 108)
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":107,"threads_started":1}
W20260620 19:10:35.778896 18711 heartbeater.cc:644] Failed to heartbeat to 127.9.155.126:40955 (0 consecutive failures): Not authorized: Failed to ping master at 127.9.155.126:40955: Client connection negotiation failed: client connection to 127.9.155.126:40955: Server oryx/127.9.155.126@KRBTEST.COM not found in Kerberos database
Jun 20 19:10:35 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.65@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:10:35 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.65@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:10:35.785228 18714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:35.779372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40955 (local address 127.9.155.65:56965)
0620 19:10:35.779525 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:35.779542 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:35.779639 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:35.779969 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:35.779972 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:35.780154 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:35.780310 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:35.780318 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:35.781122 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:35.781126 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:35.782023 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:35.782036 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:35.782489 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:35.782492 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:35.782590 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:35.783097 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:35.783111 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:35.785095 (+ 1984us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.155.126:40955: Server oryx/127.9.155.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":60}
W20260620 19:10:35.785351 18713 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:10:35.779457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.65:56965 (local address 127.9.155.126:40955)
0620 19:10:35.779564 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:35.779567 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:35.779585 (+ 18us) server_negotiation.cc:407] Connection header received
0620 19:10:35.779781 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:35.779784 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:35.779838 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:35.779924 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:35.780420 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:35.781009 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:35.782160 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:35.782374 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:35.782459 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:35.785248 (+ 2789us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.155.65:56965: BlockingRecv error: recv got EOF from 127.9.155.65:56965 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Jun 20 19:10:35 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.65@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:10:35 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.65@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:10:35.790867 18714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:35.785738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40955 (local address 127.9.155.65:45499)
0620 19:10:35.785861 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:35.785875 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:35.785950 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:35.786223 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:35.786226 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:35.786392 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:35.786496 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:35.786502 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:35.787193 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:35.787196 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:35.788021 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:35.788032 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:35.788523 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:35.788526 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:35.788611 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:35.789002 (+ 391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:35.789015 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:35.790741 (+ 1726us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.155.126:40955: Server oryx/127.9.155.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 19:10:35.791069 18711 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 19:10:35.791121 18713 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:10:35.785946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.65:45499 (local address 127.9.155.126:40955)
0620 19:10:35.786043 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:35.786045 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:35.786057 (+ 12us) server_negotiation.cc:407] Connection header received
0620 19:10:35.786089 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:35.786092 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:35.786136 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:35.786198 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:35.786604 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:35.787107 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:35.788142 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:35.788428 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:35.788515 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:35.790879 (+ 2364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.155.65:45499: BlockingRecv error: recv got EOF from 127.9.155.65:45499 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260620 19:10:35.884030 18715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:35.884399 18715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:35.884451 18715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:35.888733 18715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:35.888799 18715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:35.888866 18715 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.66
I20260620 19:10:35.894342 18715 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:46433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/krb5kdc/oryx_127.9.155.66.keytab
--principal=oryx/127.9.155.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.66:35221
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.155.66
--webserver_port=33141
--tserver_master_addrs=127.9.155.126:40955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.66
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:10:35.895444 18715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:35.896488 18715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:10:35.906435 18720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:35.906533 18715 server_base.cc:1034] running on GCE node
W20260620 19:10:35.906543 18723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:35.906445 18721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:35.907131 18715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:35.907825 18715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:35.908972 18715 hybrid_clock.cc:648] HybridClock initialized: now 1781982635908948 us; error 40 us; skew 500 ppm
Jun 20 19:10:35 dist-test-slave-n3fx krb5kdc[17980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982635, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.155.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:35.911631 18715 init.cc:377] Logged in from keytab as oryx/127.9.155.66@KRBTEST.COM (short username oryx)
I20260620 19:10:35.912655 18715 webserver.cc:460] Webserver started at http://127.9.155.66:33141/ using document root <none> and password file <none>
I20260620 19:10:35.913208 18715 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:35.913281 18715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:35.917132 18715 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:10:35.919478 18730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:35.920936 18715 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:10:35.921061 18715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-1/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-1/wal
uuid: "140a6670e6f74a5d9ed996a741cc352b"
format_stamp: "Formatted at 2026-06-20 19:10:35 on dist-test-slave-n3fx"
server_key: "4f1a8bc1996986bc800d5a1b763ce618"
server_key_iv: "f8188a6bee05d3320a172b547a906fa2"
server_key_version: "encryptionkey@0"
I20260620 19:10:35.921628 18715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:35.934625 18715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:35.937860 18715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:35.938077 18715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:35.938668 18715 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:10:35.939498 18715 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:10:35.939543 18715 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:35.939584 18715 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:10:35.939612 18715 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:35.948314 18715 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.66:35221
I20260620 19:10:35.948347 18843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.66:35221 every 8 connection(s)
I20260620 19:10:35.949328 18715 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
Jun 20 19:10:35 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.66@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:10:35.958987 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 18715
Jun 20 19:10:35 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.66@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:10:35.959414 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.67:33055
--local_ip_for_outbound_sockets=127.9.155.67
--tserver_master_addrs=127.9.155.126:40955
--webserver_port=39813
--webserver_interface=127.9.155.67
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.67.keytab
--principal=kudu/127.9.155.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:46433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.9.155.67
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/krb5kdc/oryx_127.9.155.67.keytab with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:10:35.960644 18713 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:10:35.951210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.66:43617 (local address 127.9.155.126:40955)
0620 19:10:35.951395 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:35.951400 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:35.952525 (+ 1125us) server_negotiation.cc:407] Connection header received
0620 19:10:35.953383 (+ 858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:35.953387 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:35.953438 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:35.953513 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:35.954756 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:35.955308 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:35.956568 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:35.956793 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:35.956893 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:35.960530 (+ 3637us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.155.66:43617: BlockingRecv error: recv got EOF from 127.9.155.66:43617 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 19:10:35.961297 18846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:35.951626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40955 (local address 127.9.155.66:43617)
0620 19:10:35.952341 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:35.952440 (+ 99us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:35.953191 (+ 751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:35.953669 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:35.953676 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:35.954047 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:35.954595 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:35.954612 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:35.955466 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:35.955469 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:35.956439 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:35.956448 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:35.956947 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:35.956950 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:35.957104 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:35.957717 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:35.957733 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:35.960327 (+ 2594us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.155.126:40955: Server oryx/127.9.155.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":128,"threads_started":1}
W20260620 19:10:35.961777 18844 heartbeater.cc:644] Failed to heartbeat to 127.9.155.126:40955 (0 consecutive failures): Not authorized: Failed to ping master at 127.9.155.126:40955: Client connection negotiation failed: client connection to 127.9.155.126:40955: Server oryx/127.9.155.126@KRBTEST.COM not found in Kerberos database
Jun 20 19:10:35 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.66@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:10:35 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.66@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:10:35.967320 18846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:35.962211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40955 (local address 127.9.155.66:36985)
0620 19:10:35.962308 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:35.962322 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:35.962403 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:35.962617 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:35.962620 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:35.962775 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:35.962878 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:35.962883 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:35.963667 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:35.963670 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:35.964247 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:35.964258 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:35.964669 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:35.964673 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:35.964759 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:35.965275 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:35.965286 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:35.967191 (+ 1905us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.155.126:40955: Server oryx/127.9.155.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":28}
W20260620 19:10:35.967422 18713 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:10:35.962300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.66:36985 (local address 127.9.155.126:40955)
0620 19:10:35.962419 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:35.962423 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:35.962439 (+ 16us) server_negotiation.cc:407] Connection header received
0620 19:10:35.962481 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:35.962484 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:35.962526 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:35.962589 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:35.962989 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:35.963564 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:35.964358 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:35.964572 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:35.964659 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:35.967319 (+ 2660us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.155.66:36985: BlockingRecv error: recv got EOF from 127.9.155.66:36985 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Jun 20 19:10:35 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.66@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:10:35 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.66@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:10:35.972786 18846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:35.967924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40955 (local address 127.9.155.66:55621)
0620 19:10:35.968149 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:35.968162 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:35.968262 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:35.968470 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:35.968472 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:35.968617 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:35.968685 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:35.968689 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:35.969368 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:35.969371 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:35.969884 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:35.969890 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:35.970277 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:35.970281 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:35.970363 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:35.970866 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:35.970877 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:35.972657 (+ 1780us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.155.126:40955: Server oryx/127.9.155.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163}
W20260620 19:10:35.973057 18713 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:10:35.968038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.66:55621 (local address 127.9.155.126:40955)
0620 19:10:35.968150 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:35.968154 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:35.968203 (+ 49us) server_negotiation.cc:407] Connection header received
0620 19:10:35.968343 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:35.968346 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:35.968386 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:35.968446 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:35.968782 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:35.969269 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:35.969982 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:35.970187 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:35.970270 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:35.972903 (+ 2633us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.155.66:55621: BlockingRecv error: recv got EOF from 127.9.155.66:55621 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260620 19:10:35.973207 18844 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 19:10:36.070726 18847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:36.071008 18847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:36.071051 18847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:36.075121 18847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:36.075191 18847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:36.075250 18847 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.67
I20260620 19:10:36.080675 18847 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:46433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/krb5kdc/oryx_127.9.155.67.keytab
--principal=oryx/127.9.155.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.67:33055
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.155.67
--webserver_port=39813
--tserver_master_addrs=127.9.155.126:40955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18847
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.67
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:10:36.081737 18847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:36.082718 18847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:10:36.092252 18852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:36.092269 18853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:36.092435 18847 server_base.cc:1034] running on GCE node
W20260620 19:10:36.092274 18855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:36.092938 18847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:36.093577 18847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:36.094735 18847 hybrid_clock.cc:648] HybridClock initialized: now 1781982636094714 us; error 33 us; skew 500 ppm
Jun 20 19:10:36 dist-test-slave-n3fx krb5kdc[17980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982636, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.155.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:36.097498 18847 init.cc:377] Logged in from keytab as oryx/127.9.155.67@KRBTEST.COM (short username oryx)
I20260620 19:10:36.098379 18847 webserver.cc:460] Webserver started at http://127.9.155.67:39813/ using document root <none> and password file <none>
I20260620 19:10:36.098919 18847 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:36.099002 18847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:36.102949 18847 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 19:10:36.105258 18862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:36.106850 18847 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:10:36.106997 18847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-2/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-2/wal
uuid: "e7bd50dbb9794f36a99b908294d57efd"
format_stamp: "Formatted at 2026-06-20 19:10:35 on dist-test-slave-n3fx"
server_key: "eb9dde68e6851216aa264a82734a494a"
server_key_iv: "4added1cc4efa6da9ae0e8937be653ce"
server_key_version: "encryptionkey@0"
I20260620 19:10:36.107632 18847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:36.124650 18847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:36.127707 18847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:36.127877 18847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:36.128492 18847 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:10:36.129303 18847 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:10:36.129351 18847 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:36.129405 18847 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:10:36.129429 18847 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:36.137876 18847 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.67:33055
I20260620 19:10:36.137917 18975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.67:33055 every 8 connection(s)
I20260620 19:10:36.138856 18847 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
I20260620 19:10:36.146395 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 18847
Jun 20 19:10:36 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.67@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:10:36 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.67@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
W20260620 19:10:36.150417 18713 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:10:36.140764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.67:37539 (local address 127.9.155.126:40955)
0620 19:10:36.140901 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:36.140905 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:36.142112 (+ 1207us) server_negotiation.cc:407] Connection header received
0620 19:10:36.143087 (+ 975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:36.143091 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:36.143150 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:36.143246 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:36.144587 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:36.145394 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:36.146614 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:36.146844 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:36.146951 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:36.150291 (+ 3340us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.155.67:37539: BlockingRecv error: recv got EOF from 127.9.155.67:37539 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 19:10:36.151010 18978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:36.141207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40955 (local address 127.9.155.67:37539)
0620 19:10:36.141941 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:36.142010 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:36.142859 (+ 849us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:36.143403 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:36.143412 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:36.143836 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:36.144435 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:36.144450 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:36.145520 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:36.145523 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:36.146469 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:36.146480 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:36.147024 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:36.147028 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:36.147248 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:36.147849 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:36.147869 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:36.150157 (+ 2288us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.155.126:40955: Server oryx/127.9.155.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":149,"threads_started":1}
W20260620 19:10:36.151499 18976 heartbeater.cc:644] Failed to heartbeat to 127.9.155.126:40955 (0 consecutive failures): Not authorized: Failed to ping master at 127.9.155.126:40955: Client connection negotiation failed: client connection to 127.9.155.126:40955: Server oryx/127.9.155.126@KRBTEST.COM not found in Kerberos database
Jun 20 19:10:36 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.67@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:10:36.156352 18980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:36.147880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32944 (local address 127.9.155.126:40955)
0620 19:10:36.148131 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:36.148134 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:36.148146 (+ 12us) server_negotiation.cc:407] Connection header received
0620 19:10:36.148176 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:36.148178 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:36.148239 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:36.148335 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:36.149055 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:36.149610 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:36.150594 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:36.150823 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:36.150908 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:36.152497 (+ 1589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:36.152506 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:36.152527 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:36.152555 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:36.154245 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:36.154681 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:36.154688 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:36.154694 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:36.154825 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:36.155096 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:36.155099 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:36.155101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:36.155461 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:36.155650 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:36.155884 (+ 234us) server_negotiation.cc:299] Negotiation successful
0620 19:10:36.156152 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":88,"threads_started":1}
Jun 20 19:10:36 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.67@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:10:36.157135 18978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:36.151916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40955 (local address 127.9.155.67:51907)
0620 19:10:36.152145 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:36.152158 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:36.152266 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:36.152503 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:36.152506 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:36.152661 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:36.152752 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:36.152758 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:36.153646 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:36.153649 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:36.154219 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:36.154228 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:36.154671 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:36.154675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:36.154752 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:36.155134 (+ 382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:36.155145 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:36.157025 (+ 1880us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.155.126:40955: Server oryx/127.9.155.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167}
W20260620 19:10:36.157248 18713 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:10:36.152027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.67:51907 (local address 127.9.155.126:40955)
0620 19:10:36.152126 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:36.152130 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:36.152204 (+ 74us) server_negotiation.cc:407] Connection header received
0620 19:10:36.152376 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:36.152379 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:36.152416 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:36.152475 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:36.152855 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:36.153540 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:36.154337 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:36.154567 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:36.154643 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:36.157149 (+ 2506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.155.67:51907: BlockingRecv error: recv got EOF from 127.9.155.67:51907 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Jun 20 19:10:36 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.67@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:10:36 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.67@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:10:36.163059 18978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:36.157752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40955 (local address 127.9.155.67:41483)
0620 19:10:36.157907 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:36.157920 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:36.157985 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:36.158200 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:36.158203 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:36.158354 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:36.158423 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:36.158427 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:36.159247 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:36.159250 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:36.160072 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:36.160085 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:36.160674 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:36.160678 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:36.160742 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:36.161186 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:36.161196 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:36.162962 (+ 1766us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.155.126:40955: Server oryx/127.9.155.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":92}
W20260620 19:10:36.163224 18713 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:10:36.157809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.67:41483 (local address 127.9.155.126:40955)
0620 19:10:36.157939 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:36.157943 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:36.157959 (+ 16us) server_negotiation.cc:407] Connection header received
0620 19:10:36.158077 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:36.158080 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:36.158116 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:36.158174 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:36.158520 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:36.159141 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:36.160281 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:36.160539 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:36.160651 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:36.163108 (+ 2457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.155.67:41483: BlockingRecv error: recv got EOF from 127.9.155.67:41483 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260620 19:10:36.163309 18976 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 20 19:10:36 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.65@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:10:36 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.65@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:10:36.797478 18981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:36.791738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40955 (local address 127.9.155.65:50953)
0620 19:10:36.792015 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:36.792029 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:36.792128 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:36.792447 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:36.792450 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:36.792622 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:36.792742 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:36.792748 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:36.793562 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:36.793566 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:36.794162 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:36.794171 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:36.794615 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:36.794618 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:36.794698 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:36.795291 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:36.795304 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:36.797316 (+ 2012us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.155.126:40955: Server oryx/127.9.155.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":84,"threads_started":1}
W20260620 19:10:36.797577 18982 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:10:36.791856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.65:50953 (local address 127.9.155.126:40955)
0620 19:10:36.792083 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:36.792086 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:36.792103 (+ 17us) server_negotiation.cc:407] Connection header received
0620 19:10:36.792255 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:36.792259 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:36.792307 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:36.792412 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:36.792885 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:36.793427 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:36.794257 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:36.794500 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:36.794573 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:36.797463 (+ 2890us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.155.65:50953: BlockingRecv error: recv got EOF from 127.9.155.65:50953 (error 108)
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":58,"threads_started":1}
Jun 20 19:10:36 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.66@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:10:36 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.66@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:10:36.985165 18983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:36.973952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40955 (local address 127.9.155.66:49077)
0620 19:10:36.974238 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:36.974253 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:36.974383 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:36.974623 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:36.974625 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:36.974818 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:36.974933 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:36.974940 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:36.975761 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:36.975765 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:36.976612 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:36.976622 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:36.977234 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:36.977238 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:36.977329 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:36.977872 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:36.977894 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:36.984991 (+ 7097us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.155.126:40955: Server oryx/127.9.155.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":83,"threads_started":1}
W20260620 19:10:36.985252 18982 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:10:36.974102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.66:49077 (local address 127.9.155.126:40955)
0620 19:10:36.974240 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:36.974244 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:36.974390 (+ 146us) server_negotiation.cc:407] Connection header received
0620 19:10:36.974473 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:36.974477 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:36.974523 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:36.974592 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:36.975059 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:36.975639 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:36.976732 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:36.977076 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:36.977186 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:36.985141 (+ 7955us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.155.66:49077: BlockingRecv error: recv got EOF from 127.9.155.66:49077 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Jun 20 19:10:37 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.67@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:10:37 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.67@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:10:37.170787 18984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:37.164129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40955 (local address 127.9.155.67:49729)
0620 19:10:37.164440 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:37.164454 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:37.164568 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:37.164953 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:37.164955 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:37.165128 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:37.165228 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:37.165233 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:37.166167 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:37.166171 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:37.167061 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:37.167074 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:37.167542 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:37.167545 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:37.167706 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:37.168385 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:37.168402 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:37.170642 (+ 2240us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.155.126:40955: Server oryx/127.9.155.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":92,"threads_started":1}
W20260620 19:10:37.170977 18982 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:10:37.164244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.67:49729 (local address 127.9.155.126:40955)
0620 19:10:37.164399 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:37.164405 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:37.164532 (+ 127us) server_negotiation.cc:407] Connection header received
0620 19:10:37.164649 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:37.164653 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:37.164737 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:37.164885 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:37.165344 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:37.166039 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:37.167185 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:37.167426 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:37.167524 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:37.170821 (+ 3297us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.155.67:49729: BlockingRecv error: recv got EOF from 127.9.155.67:49729 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 20 19:10:37 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.65@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:10:37 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.65@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:10:37.803808 18985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:37.798293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40955 (local address 127.9.155.65:54473)
0620 19:10:37.798628 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:37.798642 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:37.798743 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:37.799011 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:37.799013 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:37.799182 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:37.799284 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:37.799290 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:37.800003 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:37.800007 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:37.800645 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:37.800655 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:37.801113 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:37.801116 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:37.801179 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:37.801631 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:37.801643 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:37.803647 (+ 2004us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.155.126:40955: Server oryx/127.9.155.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":85,"threads_started":1}
W20260620 19:10:37.803889 18986 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:10:37.798427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.65:54473 (local address 127.9.155.126:40955)
0620 19:10:37.798696 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:37.798700 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:37.798716 (+ 16us) server_negotiation.cc:407] Connection header received
0620 19:10:37.798850 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:37.798853 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:37.798899 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:37.798998 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:37.799376 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:37.799896 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:37.800771 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:37.801018 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:37.801104 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:37.803778 (+ 2674us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.155.65:54473: BlockingRecv error: recv got EOF from 127.9.155.65:54473 (error 108)
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":104,"threads_started":1}
Jun 20 19:10:37 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.66@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:10:37 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.66@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:10:37.992676 18987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:37.986197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40955 (local address 127.9.155.66:35123)
0620 19:10:37.986529 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:37.986546 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:37.986676 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:37.986970 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:37.986973 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:37.987172 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:37.987291 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:37.987299 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:37.988298 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:37.988302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:37.989175 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:37.989190 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:37.989725 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:37.989729 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:37.989803 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:37.990489 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:37.990503 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:37.992490 (+ 1987us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.155.126:40955: Server oryx/127.9.155.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":103,"threads_started":1}
W20260620 19:10:37.992746 18986 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:10:37.986349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.66:35123 (local address 127.9.155.126:40955)
0620 19:10:37.986594 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:37.986599 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:37.986619 (+ 20us) server_negotiation.cc:407] Connection header received
0620 19:10:37.986775 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:37.986778 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:37.986840 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:37.986932 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:37.987445 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:37.988139 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:37.989312 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:37.989567 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:37.989662 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:37.992645 (+ 2983us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.155.66:35123: BlockingRecv error: recv got EOF from 127.9.155.66:35123 (error 108)
Metrics: {"server-negotiator.queue_time_us":134}
Jun 20 19:10:38 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.67@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:10:38 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.67@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:10:38.177139 18988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:38.171819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40955 (local address 127.9.155.67:56093)
0620 19:10:38.172075 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:38.172093 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:38.172204 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:38.172494 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:38.172497 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:38.172687 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:38.172783 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:38.172789 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:38.173574 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:38.173578 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:38.174138 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:38.174147 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:38.174574 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:38.174578 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:38.174674 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:38.175168 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:38.175180 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:38.176961 (+ 1781us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.155.126:40955: Server oryx/127.9.155.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":69,"threads_started":1}
W20260620 19:10:38.177352 18986 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:10:38.172002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.67:56093 (local address 127.9.155.126:40955)
0620 19:10:38.172153 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:38.172157 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:38.172172 (+ 15us) server_negotiation.cc:407] Connection header received
0620 19:10:38.172330 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:38.172333 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:38.172381 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:38.172446 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:38.172901 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:38.173455 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:38.174255 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:38.174472 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:38.174558 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:38.177146 (+ 2588us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.155.67:56093: BlockingRecv error: recv got EOF from 127.9.155.67:56093 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Jun 20 19:10:38 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.65@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:10:38 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.65@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:10:38.809820 18989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:38.804566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40955 (local address 127.9.155.65:44463)
0620 19:10:38.804814 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:38.804827 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:38.804938 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:38.805231 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:38.805235 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:38.805437 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:38.805540 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:38.805546 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:38.806328 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:38.806331 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:38.806912 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:38.806920 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:38.807348 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:38.807350 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:38.807411 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:38.807862 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:38.807872 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:38.809651 (+ 1779us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.155.126:40955: Server oryx/127.9.155.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":74,"threads_started":1}
W20260620 19:10:38.809913 18990 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:10:38.804662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.65:44463 (local address 127.9.155.126:40955)
0620 19:10:38.804904 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:38.804907 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:38.804921 (+ 14us) server_negotiation.cc:407] Connection header received
0620 19:10:38.805036 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:38.805039 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:38.805080 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:38.805157 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:38.805666 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:38.806213 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:38.807042 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:38.807264 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:38.807338 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:38.809795 (+ 2457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.155.65:44463: BlockingRecv error: recv got EOF from 127.9.155.65:44463 (error 108)
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":91,"threads_started":1}
Jun 20 19:10:38 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.66@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:10:38 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.66@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:10:38.998570 18991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:38.993428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40955 (local address 127.9.155.66:41425)
0620 19:10:38.993685 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:38.993698 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:38.993796 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:38.994014 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:38.994015 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:38.994183 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:38.994274 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:38.994279 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:38.995105 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:38.995109 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:38.995663 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:38.995672 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:38.996105 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:38.996108 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:38.996172 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:38.996649 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:38.996660 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:38.998446 (+ 1786us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.155.126:40955: Server oryx/127.9.155.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":90,"threads_started":1}
W20260620 19:10:38.998739 18990 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:10:38.993471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.66:41425 (local address 127.9.155.126:40955)
0620 19:10:38.993620 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:38.993625 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:38.993829 (+ 204us) server_negotiation.cc:407] Connection header received
0620 19:10:38.993877 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:38.993879 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:38.993930 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:38.994004 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:38.994412 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:38.994969 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:38.995772 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:38.996008 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:38.996087 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:38.998602 (+ 2515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.155.66:41425: BlockingRecv error: recv got EOF from 127.9.155.66:41425 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 20 19:10:39 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.67@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:10:39 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.67@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:10:39.183841 18992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:39.177905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40955 (local address 127.9.155.67:39499)
0620 19:10:39.178166 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:39.178179 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:39.178278 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:39.178599 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:39.178602 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:39.178777 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:39.178898 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:39.178905 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:39.180002 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:39.180005 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:39.180638 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:39.180649 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:39.181240 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:39.181243 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:39.181310 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:39.181787 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:39.181799 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:39.183716 (+ 1917us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.155.126:40955: Server oryx/127.9.155.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":68,"threads_started":1}
W20260620 19:10:39.184015 18990 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:10:39.177984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.67:39499 (local address 127.9.155.126:40955)
0620 19:10:39.178130 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:39.178135 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:39.178243 (+ 108us) server_negotiation.cc:407] Connection header received
0620 19:10:39.178377 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:39.178380 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:39.178442 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:39.178572 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:39.179001 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:39.179853 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:39.180759 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:39.181109 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:39.181217 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:39.183899 (+ 2682us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.155.67:39499: BlockingRecv error: recv got EOF from 127.9.155.67:39499 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 20 19:10:39 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.65@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:10:39 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.65@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:10:39.816087 18993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:39.810716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40955 (local address 127.9.155.65:52303)
0620 19:10:39.810959 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:39.810973 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:39.811102 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:39.811376 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:39.811379 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:39.811564 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:39.811733 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:39.811744 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:39.812601 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:39.812603 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:39.813196 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:39.813205 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:39.813693 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:39.813695 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:39.813755 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:39.814220 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:39.814231 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:39.815948 (+ 1717us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.155.126:40955: Server oryx/127.9.155.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":73,"threads_started":1}
W20260620 19:10:39.816282 18994 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:10:39.810791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.65:52303 (local address 127.9.155.126:40955)
0620 19:10:39.811036 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:39.811041 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:39.811054 (+ 13us) server_negotiation.cc:407] Connection header received
0620 19:10:39.811192 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:39.811194 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:39.811248 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:39.811353 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:39.811855 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:39.812519 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:39.813303 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:39.813592 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:39.813694 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:39.816109 (+ 2415us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.155.65:52303: BlockingRecv error: recv got EOF from 127.9.155.65:52303 (error 108)
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":63,"threads_started":1}
Jun 20 19:10:39 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.66@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:10:40 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.66@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:10:40.005642 18995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:39.999358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40955 (local address 127.9.155.66:33113)
0620 19:10:39.999663 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:39.999678 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:39.999782 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:40.000134 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:40.000137 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:40.000372 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:40.000496 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:40.000504 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:40.001330 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:40.001333 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:40.001915 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:40.001923 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:40.002419 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:40.002422 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:40.002534 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:40.003007 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:40.003019 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:40.005509 (+ 2490us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.155.126:40955: Server oryx/127.9.155.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":108,"threads_started":1}
W20260620 19:10:40.005946 18994 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:10:39.999486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.66:33113 (local address 127.9.155.126:40955)
0620 19:10:39.999623 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:39.999627 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:39.999752 (+ 125us) server_negotiation.cc:407] Connection header received
0620 19:10:39.999875 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:39.999877 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:39.999924 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:40.000004 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:40.000636 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:40.001219 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:40.002037 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:40.002260 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:40.002343 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:40.005806 (+ 3463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.155.66:33113: BlockingRecv error: recv got EOF from 127.9.155.66:33113 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 20 19:10:40 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.67@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:10:40 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.67@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:10:40.193612 18996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:40.184683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40955 (local address 127.9.155.67:41471)
0620 19:10:40.184981 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:40.184995 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:40.185100 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:40.185479 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:40.185482 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:40.185663 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:40.185760 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:40.185767 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:40.186627 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:40.186630 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:40.187214 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:40.187223 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:40.187755 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:40.187758 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:40.187846 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:40.191539 (+ 3693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:40.191553 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:40.193490 (+ 1937us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.155.126:40955: Server oryx/127.9.155.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":92,"threads_started":1}
W20260620 19:10:40.193786 18994 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:10:40.184789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.67:41471 (local address 127.9.155.126:40955)
0620 19:10:40.184967 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:40.184971 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:40.185063 (+ 92us) server_negotiation.cc:407] Connection header received
0620 19:10:40.185221 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:40.185224 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:40.185326 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:40.185408 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:40.185913 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:40.186508 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:40.187374 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:40.187621 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:40.187711 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:40.193658 (+ 5947us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.155.67:41471: BlockingRecv error: recv got EOF from 127.9.155.67:41471 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Jun 20 19:10:40 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.65@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:10:40 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.65@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:10:40.823029 18997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:40.817012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40955 (local address 127.9.155.65:50789)
0620 19:10:40.817313 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:40.817385 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:40.817499 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:40.817747 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:40.817749 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:40.817952 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:40.818048 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:40.818055 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:40.818837 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:40.818841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:40.819610 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:40.819629 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:40.820169 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:40.820173 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:40.820309 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:40.820964 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:40.820979 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:40.822878 (+ 1899us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.155.126:40955: Server oryx/127.9.155.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":89,"threads_started":1}
W20260620 19:10:40.823139 18998 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:10:40.817119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.65:50789 (local address 127.9.155.126:40955)
0620 19:10:40.817346 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:40.817349 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:40.817463 (+ 114us) server_negotiation.cc:407] Connection header received
0620 19:10:40.817577 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:40.817580 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:40.817625 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:40.817706 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:40.818165 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:40.818715 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:40.819744 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:40.820055 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:40.820150 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:40.823025 (+ 2875us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.155.65:50789: BlockingRecv error: recv got EOF from 127.9.155.65:50789 (error 108)
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":57,"threads_started":1}
Jun 20 19:10:41 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.66@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:10:41 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.66@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:10:41.012295 18999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:41.006573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40955 (local address 127.9.155.66:54317)
0620 19:10:41.006886 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:41.006900 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:41.007000 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:41.007278 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:41.007281 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:41.007456 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:41.007600 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:41.007606 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:41.008453 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:41.008456 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:41.009173 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:41.009184 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:41.009644 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:41.009647 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:41.009716 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:41.010164 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:41.010175 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:41.012121 (+ 1946us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.155.126:40955: Server oryx/127.9.155.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":103,"threads_started":1}
W20260620 19:10:41.012365 18998 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:10:41.006743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.66:54317 (local address 127.9.155.126:40955)
0620 19:10:41.006888 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:41.006892 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:41.006965 (+ 73us) server_negotiation.cc:407] Connection header received
0620 19:10:41.007077 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:41.007079 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:41.007122 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:41.007199 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:41.007730 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:41.008332 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:41.009285 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:41.009543 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:41.009646 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:41.012267 (+ 2621us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.155.66:54317: BlockingRecv error: recv got EOF from 127.9.155.66:54317 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 20 19:10:41 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.67@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:10:41 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.67@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:10:41.199709 19000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:41.194390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40955 (local address 127.9.155.67:41451)
0620 19:10:41.194656 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:41.194668 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:41.194764 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:41.194989 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:41.194992 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:41.195166 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:41.195258 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:41.195262 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:41.195963 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:41.195965 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:41.196560 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:41.196568 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:41.197019 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:41.197022 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:41.197092 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:41.197556 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:41.197566 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:41.199562 (+ 1996us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.155.126:40955: Server oryx/127.9.155.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":85,"threads_started":1}
W20260620 19:10:41.199813 18998 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:10:41.194491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.67:41451 (local address 127.9.155.126:40955)
0620 19:10:41.194638 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:41.194641 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:41.194732 (+ 91us) server_negotiation.cc:407] Connection header received
0620 19:10:41.194840 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:41.194843 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:41.194892 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:41.194960 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:41.195353 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:41.195865 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:41.196648 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:41.196860 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:41.196959 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:41.199708 (+ 2749us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.155.67:41451: BlockingRecv error: recv got EOF from 127.9.155.67:41451 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 20 19:10:41 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.65@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:10:41 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.65@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:10:41.829326 19001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:41.823846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40955 (local address 127.9.155.65:39455)
0620 19:10:41.824124 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:41.824137 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:41.824261 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:41.824488 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:41.824490 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:41.824709 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:41.824805 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:41.824811 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:41.825615 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:41.825619 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:41.826403 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:41.826413 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:41.826869 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:41.826871 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:41.826940 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:41.827474 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:41.827486 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:41.829210 (+ 1724us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.155.126:40955: Server oryx/127.9.155.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":95,"threads_started":1}
W20260620 19:10:41.829459 19002 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:10:41.823955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.65:39455 (local address 127.9.155.126:40955)
0620 19:10:41.824208 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:41.824232 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:41.824250 (+ 18us) server_negotiation.cc:407] Connection header received
0620 19:10:41.824341 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:41.824343 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:41.824389 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:41.824464 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:41.824921 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:41.825511 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:41.826526 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:41.826783 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:41.826857 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:41.829336 (+ 2479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.155.65:39455: BlockingRecv error: recv got EOF from 127.9.155.65:39455 (error 108)
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":66,"threads_started":1}
Jun 20 19:10:42 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.66@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:10:42 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.66@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:10:42.018776 19003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:42.013174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40955 (local address 127.9.155.66:48395)
0620 19:10:42.013446 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:42.013466 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:42.013606 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:42.014000 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:42.014003 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:42.014194 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:42.014291 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:42.014297 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:42.015058 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:42.015061 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:42.015651 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:42.015659 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:42.016186 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:42.016190 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:42.016314 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:42.016865 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:42.016878 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:42.018660 (+ 1782us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.155.126:40955: Server oryx/127.9.155.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":70,"threads_started":1}
W20260620 19:10:42.019032 19002 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:10:42.013242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.66:48395 (local address 127.9.155.126:40955)
0620 19:10:42.013375 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:42.013380 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:42.013559 (+ 179us) server_negotiation.cc:407] Connection header received
0620 19:10:42.013736 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:42.013739 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:42.013794 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:42.013877 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:42.014417 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:42.014955 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:42.015781 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:42.016059 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:42.016162 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:42.018794 (+ 2632us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.155.66:48395: BlockingRecv error: recv got EOF from 127.9.155.66:48395 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Jun 20 19:10:42 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.67@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:10:42 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.67@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:10:42.206516 19004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:42.200629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40955 (local address 127.9.155.67:35501)
0620 19:10:42.200905 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:42.200918 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:42.201022 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:42.201341 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:42.201344 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:42.201708 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:42.201833 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:42.201840 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:42.202643 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:42.202646 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:42.203245 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:42.203254 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:42.203770 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:42.203773 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:42.203845 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:42.204422 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:42.204434 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:42.206396 (+ 1962us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.155.126:40955: Server oryx/127.9.155.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":77,"threads_started":1}
W20260620 19:10:42.206692 19002 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:10:42.200749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.67:35501 (local address 127.9.155.126:40955)
0620 19:10:42.200920 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:42.200924 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:42.200985 (+ 61us) server_negotiation.cc:407] Connection header received
0620 19:10:42.201161 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:42.201163 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:42.201218 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:42.201301 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:42.201987 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:42.202516 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:42.203385 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:42.203629 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:42.203723 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:42.206573 (+ 2850us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.155.67:35501: BlockingRecv error: recv got EOF from 127.9.155.67:35501 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 20 19:10:42 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.65@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:10:42 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.65@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:10:42.835814 19005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:42.830226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40955 (local address 127.9.155.65:39485)
0620 19:10:42.830516 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:42.830530 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:42.830623 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:42.830882 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:42.830884 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:42.831067 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:42.831165 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:42.831171 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:42.832074 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:42.832077 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:42.832744 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:42.832756 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:42.833248 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:42.833252 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:42.833323 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:42.833898 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:42.833910 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:42.835691 (+ 1781us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.155.126:40955: Server oryx/127.9.155.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":85,"threads_started":1}
W20260620 19:10:42.836077 19006 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:10:42.830276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.65:39485 (local address 127.9.155.126:40955)
0620 19:10:42.830523 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:42.830526 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:42.830590 (+ 64us) server_negotiation.cc:407] Connection header received
0620 19:10:42.830694 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:42.830696 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:42.830737 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:42.830825 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:42.831360 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:42.831938 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:42.832890 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:42.833129 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:42.833210 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:42.835898 (+ 2688us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.155.65:39485: BlockingRecv error: recv got EOF from 127.9.155.65:39485 (error 108)
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":74,"threads_started":1}
Jun 20 19:10:43 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.66@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:10:43 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.66@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:10:43.025393 19007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:43.019716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40955 (local address 127.9.155.66:45891)
0620 19:10:43.020064 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:43.020077 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:43.020195 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:43.020545 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:43.020548 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:43.020719 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:43.020880 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:43.020889 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:43.021792 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:43.021795 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:43.022361 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:43.022369 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:43.022785 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:43.022788 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:43.022876 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:43.023380 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:43.023392 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:43.025246 (+ 1854us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.155.126:40955: Server oryx/127.9.155.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":101,"threads_started":1}
W20260620 19:10:43.025504 19006 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:10:43.019794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.66:45891 (local address 127.9.155.126:40955)
0620 19:10:43.019997 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:43.020001 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:43.020153 (+ 152us) server_negotiation.cc:407] Connection header received
0620 19:10:43.020322 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:43.020325 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:43.020370 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:43.020502 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:43.021011 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:43.021681 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:43.022472 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:43.022687 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:43.022772 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:43.025388 (+ 2616us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.155.66:45891: BlockingRecv error: recv got EOF from 127.9.155.66:45891 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
Jun 20 19:10:43 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.67@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:10:43 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.67@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:10:43.217971 19008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:43.207416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40955 (local address 127.9.155.67:56331)
0620 19:10:43.207722 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:43.207734 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:43.207871 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:43.208107 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:43.208109 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:43.208318 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:43.208413 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:43.208419 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:43.209315 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:43.209320 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:43.209934 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:43.209946 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:43.210495 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:43.210500 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:43.210574 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:43.215770 (+ 5196us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:43.215785 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:43.217841 (+ 2056us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.155.126:40955: Server oryx/127.9.155.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":72,"threads_started":1}
W20260620 19:10:43.218246 19006 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:10:43.207495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.67:56331 (local address 127.9.155.126:40955)
0620 19:10:43.207632 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:43.207635 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:43.207849 (+ 214us) server_negotiation.cc:407] Connection header received
0620 19:10:43.207949 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:43.207952 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:43.208003 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:43.208081 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:43.208578 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:43.209122 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:43.210085 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:43.210323 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:43.210425 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:43.218033 (+ 7608us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.155.67:56331: BlockingRecv error: recv got EOF from 127.9.155.67:56331 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Jun 20 19:10:43 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.65@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:10:43 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.65@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:10:43.842476 19009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:43.836687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40955 (local address 127.9.155.65:54791)
0620 19:10:43.836950 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:43.836962 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:43.837053 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:43.837296 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:43.837298 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:43.837507 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:43.837606 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:43.837612 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:43.838500 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:43.838503 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:43.839264 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:43.839279 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:43.839763 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:43.839766 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:43.839832 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:43.840486 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:43.840498 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:43.842313 (+ 1815us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.155.126:40955: Server oryx/127.9.155.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":74,"threads_started":1}
W20260620 19:10:43.842594 19010 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:10:43.836849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.65:54791 (local address 127.9.155.126:40955)
0620 19:10:43.837080 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:43.837084 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:43.837097 (+ 13us) server_negotiation.cc:407] Connection header received
0620 19:10:43.837138 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:43.837140 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:43.837181 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:43.837271 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:43.837708 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:43.838303 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:43.839384 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:43.839662 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:43.839744 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:43.842477 (+ 2733us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.155.65:54791: BlockingRecv error: recv got EOF from 127.9.155.65:54791 (error 108)
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":75,"threads_started":1}
Jun 20 19:10:44 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.66@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:10:44 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.66@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:10:44.031692 19011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:44.026155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40955 (local address 127.9.155.66:37491)
0620 19:10:44.026505 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:44.026519 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:44.026622 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:44.026851 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:44.026853 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:44.027019 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:44.027113 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:44.027118 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:44.028131 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:44.028135 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:44.028824 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:44.028834 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:44.029258 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:44.029262 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:44.029336 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:44.029830 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:44.029841 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:44.031542 (+ 1701us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.155.126:40955: Server oryx/127.9.155.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":79,"threads_started":1}
W20260620 19:10:44.031785 19010 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:10:44.026323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.66:37491 (local address 127.9.155.126:40955)
0620 19:10:44.026500 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:44.026505 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:44.026615 (+ 110us) server_negotiation.cc:407] Connection header received
0620 19:10:44.026695 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:44.026697 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:44.026745 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:44.026824 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:44.027237 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:44.028006 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:44.028931 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:44.029153 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:44.029241 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:44.031685 (+ 2444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.155.66:37491: BlockingRecv error: recv got EOF from 127.9.155.66:37491 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Jun 20 19:10:44 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.67@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:10:44 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.67@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:10:44.224692 19012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:44.218957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40955 (local address 127.9.155.67:47645)
0620 19:10:44.219307 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:44.219323 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:44.219450 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:44.219717 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:44.219720 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:44.219926 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:44.220068 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:44.220077 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:44.220889 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:44.220892 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:44.221468 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:44.221476 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:44.221960 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:44.221964 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:44.222040 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:44.222546 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:44.222559 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:44.224537 (+ 1978us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.155.126:40955: Server oryx/127.9.155.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":73,"threads_started":1}
W20260620 19:10:44.224952 19010 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:10:44.219022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.67:47645 (local address 127.9.155.126:40955)
0620 19:10:44.219170 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:44.219174 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:44.219409 (+ 235us) server_negotiation.cc:407] Connection header received
0620 19:10:44.219536 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:44.219540 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:44.219598 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:44.219670 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:44.220188 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:44.220763 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:44.221591 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:44.221810 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:44.221918 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:44.224712 (+ 2794us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.155.67:47645: BlockingRecv error: recv got EOF from 127.9.155.67:47645 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 20 19:10:44 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.65@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:10:44 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.65@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:10:44.848578 19013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:44.843190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40955 (local address 127.9.155.65:54769)
0620 19:10:44.843443 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:44.843456 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:44.843541 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:44.843759 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:44.843761 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:44.843956 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:44.844054 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:44.844060 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:44.844850 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:44.844852 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:44.845444 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:44.845453 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:44.845990 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:44.845993 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:44.846067 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:44.846651 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:44.846663 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:44.848457 (+ 1794us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.155.126:40955: Server oryx/127.9.155.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":83,"threads_started":1}
W20260620 19:10:44.848774 19014 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:10:44.843269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.65:54769 (local address 127.9.155.126:40955)
0620 19:10:44.843523 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:44.843526 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:44.843540 (+ 14us) server_negotiation.cc:407] Connection header received
0620 19:10:44.843617 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:44.843619 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:44.843662 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:44.843739 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:44.844169 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:44.844755 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:44.845602 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:44.845859 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:44.845948 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:44.848623 (+ 2675us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.155.65:54769: BlockingRecv error: recv got EOF from 127.9.155.65:54769 (error 108)
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":83,"threads_started":1}
Jun 20 19:10:45 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.66@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:10:45 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.66@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:10:45.037845 19015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:45.032504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40955 (local address 127.9.155.66:35725)
0620 19:10:45.032812 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:45.032823 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:45.032923 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:45.033123 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:45.033125 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:45.033293 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:45.033386 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:45.033391 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:45.034159 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:45.034162 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:45.034768 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:45.034777 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:45.035220 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:45.035223 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:45.035296 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:45.035857 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:45.035870 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:45.037704 (+ 1834us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.155.126:40955: Server oryx/127.9.155.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":75,"threads_started":1}
W20260620 19:10:45.038012 19014 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:10:45.032553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.66:35725 (local address 127.9.155.126:40955)
0620 19:10:45.032680 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:45.032684 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:45.032887 (+ 203us) server_negotiation.cc:407] Connection header received
0620 19:10:45.032992 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:45.032994 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:45.033036 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:45.033101 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:45.033516 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:45.034052 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:45.034863 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:45.035094 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:45.035167 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:45.037861 (+ 2694us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.155.66:35725: BlockingRecv error: recv got EOF from 127.9.155.66:35725 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Jun 20 19:10:45 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.67@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:10:45 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.67@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:10:45.233819 19016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:45.225549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40955 (local address 127.9.155.67:40011)
0620 19:10:45.225807 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:45.225822 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:45.225936 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:45.226171 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:45.226175 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:45.226410 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:45.226523 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:45.226529 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:45.227298 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:45.227302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:45.227863 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:45.227870 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:45.228322 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:45.228325 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:45.228417 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:45.231583 (+ 3166us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:45.231596 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:45.233644 (+ 2048us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.155.126:40955: Server oryx/127.9.155.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":86,"threads_started":1}
W20260620 19:10:45.234063 19014 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:10:45.225652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.67:40011 (local address 127.9.155.126:40955)
0620 19:10:45.225784 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:45.225789 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:45.225891 (+ 102us) server_negotiation.cc:407] Connection header received
0620 19:10:45.226026 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:45.226029 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:45.226077 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:45.226140 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:45.226635 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:45.227193 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:45.227973 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:45.228192 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:45.228314 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:45.233818 (+ 5504us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.155.67:40011: BlockingRecv error: recv got EOF from 127.9.155.67:40011 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 20 19:10:45 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.65@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:10:45 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.65@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:10:45.855082 19017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:45.849459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40955 (local address 127.9.155.65:46409)
0620 19:10:45.849765 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:45.849782 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:45.849878 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:45.850106 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:45.850108 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:45.850277 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:45.850372 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:45.850377 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:45.851205 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:45.851208 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:45.851897 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:45.851911 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:45.852466 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:45.852469 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:45.852556 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:45.853070 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:45.853082 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:45.854865 (+ 1783us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.155.126:40955: Server oryx/127.9.155.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":91,"threads_started":1}
W20260620 19:10:45.855149 19018 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:10:45.849596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.65:46409 (local address 127.9.155.126:40955)
0620 19:10:45.849855 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:45.849858 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:45.849873 (+ 15us) server_negotiation.cc:407] Connection header received
0620 19:10:45.849959 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:45.849962 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:45.850001 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:45.850080 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:45.850497 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:45.851084 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:45.852024 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:45.852353 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:45.852448 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:45.855031 (+ 2583us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.155.65:46409: BlockingRecv error: recv got EOF from 127.9.155.65:46409 (error 108)
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":94,"threads_started":1}
Jun 20 19:10:46 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.66@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:10:46 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.66@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:10:46.044312 19019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:46.038677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40955 (local address 127.9.155.66:60625)
0620 19:10:46.038991 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:46.039005 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:46.039112 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:46.039356 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:46.039359 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:46.039566 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:46.039658 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:46.039664 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:46.040452 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:46.040456 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:46.041023 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:46.041030 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:46.041475 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:46.041479 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:46.041557 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:46.042085 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:46.042096 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:46.044138 (+ 2042us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.155.126:40955: Server oryx/127.9.155.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":96,"threads_started":1}
W20260620 19:10:46.044448 19018 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:10:46.038784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.66:60625 (local address 127.9.155.126:40955)
0620 19:10:46.038941 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:46.038946 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:46.039077 (+ 131us) server_negotiation.cc:407] Connection header received
0620 19:10:46.039205 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:46.039208 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:46.039252 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:46.039321 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:46.039775 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:46.040342 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:46.041134 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:46.041372 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:46.041466 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:46.044330 (+ 2864us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.155.66:60625: BlockingRecv error: recv got EOF from 127.9.155.66:60625 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 20 19:10:46 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.67@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:10:46 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.67@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:10:46.240378 19020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:46.234775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40955 (local address 127.9.155.67:40575)
0620 19:10:46.235165 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:46.235178 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:46.235294 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:46.235618 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:46.235621 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:46.235884 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:46.235987 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:46.235993 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:46.236847 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:46.236851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:46.237428 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:46.237437 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:46.237869 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:46.237873 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:46.237957 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:46.238464 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:46.238478 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:46.240190 (+ 1712us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.155.126:40955: Server oryx/127.9.155.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":124,"threads_started":1}
W20260620 19:10:46.240594 19018 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:10:46.234949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.67:40575 (local address 127.9.155.126:40955)
0620 19:10:46.235090 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:46.235094 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:46.235242 (+ 148us) server_negotiation.cc:407] Connection header received
0620 19:10:46.235420 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:46.235423 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:46.235473 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:46.235558 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:46.236103 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:46.236727 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:46.237544 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:46.237765 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:46.237854 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:46.240375 (+ 2521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.155.67:40575: BlockingRecv error: recv got EOF from 127.9.155.67:40575 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 20 19:10:46 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.65@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:10:46 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.65@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:10:46.861097 19021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:46.855852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40955 (local address 127.9.155.65:37115)
0620 19:10:46.856146 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:46.856160 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:46.856290 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:46.856539 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:46.856540 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:46.856716 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:46.856815 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:46.856821 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:46.857624 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:46.857626 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:46.858174 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:46.858181 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:46.858642 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:46.858644 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:46.858701 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:46.859210 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:46.859221 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:46.860968 (+ 1747us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.155.126:40955: Server oryx/127.9.155.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":81,"threads_started":1}
W20260620 19:10:46.861264 19022 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:10:46.855972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.65:37115 (local address 127.9.155.126:40955)
0620 19:10:46.856230 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:46.856235 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:46.856248 (+ 13us) server_negotiation.cc:407] Connection header received
0620 19:10:46.856369 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:46.856371 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:46.856413 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:46.856519 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:46.856950 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:46.857528 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:46.858281 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:46.858552 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:46.858632 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:46.861123 (+ 2491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.155.65:37115: BlockingRecv error: recv got EOF from 127.9.155.65:37115 (error 108)
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":77,"threads_started":1}
Jun 20 19:10:47 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.66@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:10:47 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.66@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:10:47.050314 19023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:47.045101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40955 (local address 127.9.155.66:60949)
0620 19:10:47.045379 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:47.045391 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:47.045510 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:47.045750 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:47.045753 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:47.045955 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:47.046057 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:47.046063 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:47.046820 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:47.046823 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:47.047390 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:47.047398 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:47.047814 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:47.047818 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:47.047893 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:47.048417 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:47.048429 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:47.050196 (+ 1767us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.155.126:40955: Server oryx/127.9.155.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":82,"threads_started":1}
W20260620 19:10:47.050554 19022 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:10:47.045198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.66:60949 (local address 127.9.155.126:40955)
0620 19:10:47.045392 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:47.045396 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:47.045466 (+ 70us) server_negotiation.cc:407] Connection header received
0620 19:10:47.045601 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:47.045605 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:47.045651 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:47.045718 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:47.046169 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:47.046711 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:47.047505 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:47.047717 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:47.047807 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:47.050421 (+ 2614us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.155.66:60949: BlockingRecv error: recv got EOF from 127.9.155.66:60949 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 20 19:10:47 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.67@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:10:47 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.67@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:10:47.246647 19024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:47.241194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40955 (local address 127.9.155.67:50195)
0620 19:10:47.241489 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:47.241503 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:47.241604 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:47.241858 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:47.241860 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:47.242057 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:47.242155 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:47.242161 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:47.243023 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:47.243027 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:47.243618 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:47.243629 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:47.244120 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:47.244123 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:47.244187 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:47.244704 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:47.244718 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:47.246525 (+ 1807us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.155.126:40955: Server oryx/127.9.155.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":78,"threads_started":1}
W20260620 19:10:47.246848 19022 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:10:47.241337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.67:50195 (local address 127.9.155.126:40955)
0620 19:10:47.241508 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:47.241511 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:47.241566 (+ 55us) server_negotiation.cc:407] Connection header received
0620 19:10:47.241683 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:47.241685 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:47.241732 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:47.241820 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:47.242262 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:47.242847 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:47.243755 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:47.244010 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:47.244096 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:47.246708 (+ 2612us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.155.67:50195: BlockingRecv error: recv got EOF from 127.9.155.67:50195 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Jun 20 19:10:47 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.65@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:10:47 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.65@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:10:47.867520 19025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:47.861893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40955 (local address 127.9.155.65:49765)
0620 19:10:47.862175 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:47.862189 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:47.862285 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:47.862558 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:47.862560 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:47.862732 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:47.862829 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:47.862834 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:47.863615 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:47.863618 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:47.864254 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:47.864264 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:47.864823 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:47.864826 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:47.864898 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:47.865387 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:47.865399 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:47.867394 (+ 1995us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.155.126:40955: Server oryx/127.9.155.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":81,"threads_started":1}
W20260620 19:10:47.867849 19026 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:10:47.862057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.65:49765 (local address 127.9.155.126:40955)
0620 19:10:47.862289 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:47.862292 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:47.862306 (+ 14us) server_negotiation.cc:407] Connection header received
0620 19:10:47.862345 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:47.862347 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:47.862387 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:47.862504 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:47.862920 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:47.863490 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:47.864444 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:47.864695 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:47.864784 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:47.867582 (+ 2798us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.155.65:49765: BlockingRecv error: recv got EOF from 127.9.155.65:49765 (error 108)
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":92,"threads_started":1}
Jun 20 19:10:48 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.66@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:10:48 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.66@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:10:48.056787 19027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:48.051081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40955 (local address 127.9.155.66:36529)
0620 19:10:48.051331 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:48.051346 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:48.051468 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:48.051749 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:48.051752 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:48.051968 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:48.052071 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:48.052077 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:48.052910 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:48.052913 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:48.053500 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:48.053509 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:48.053982 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:48.053985 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:48.054053 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:48.054576 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:48.054588 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:48.056658 (+ 2070us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.155.126:40955: Server oryx/127.9.155.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":70,"threads_started":1}
W20260620 19:10:48.057003 19026 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:10:48.051134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.66:36529 (local address 127.9.155.126:40955)
0620 19:10:48.051273 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:48.051277 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:48.051419 (+ 142us) server_negotiation.cc:407] Connection header received
0620 19:10:48.051552 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:48.051555 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:48.051600 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:48.051674 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:48.052167 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:48.052784 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:48.053640 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:48.053868 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:48.053961 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:48.056848 (+ 2887us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.155.66:36529: BlockingRecv error: recv got EOF from 127.9.155.66:36529 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 20 19:10:48 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.67@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:10:48 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.67@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:10:48.252987 19028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:48.247544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40955 (local address 127.9.155.67:37003)
0620 19:10:48.247812 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:48.247826 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:48.247931 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:48.248194 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:48.248196 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:48.248398 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:48.248497 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:48.248503 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:48.249312 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:48.249315 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:48.249869 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:48.249877 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:48.250387 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:48.250390 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:48.250462 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:48.251023 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:48.251037 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:48.252871 (+ 1834us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.155.126:40955: Server oryx/127.9.155.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":85,"threads_started":1}
W20260620 19:10:48.253146 19026 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:10:48.247657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.67:37003 (local address 127.9.155.126:40955)
0620 19:10:48.247821 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:48.247826 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:48.247893 (+ 67us) server_negotiation.cc:407] Connection header received
0620 19:10:48.248016 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:48.248019 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:48.248077 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:48.248144 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:48.248625 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:48.249198 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:48.250000 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:48.250224 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:48.250315 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:48.253021 (+ 2706us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.155.67:37003: BlockingRecv error: recv got EOF from 127.9.155.67:37003 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Jun 20 19:10:48 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.65@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:10:48 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.65@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:10:48.874223 19029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:48.868329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40955 (local address 127.9.155.65:41109)
0620 19:10:48.868594 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:48.868606 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:48.868699 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:48.869021 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:48.869023 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:48.869212 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:48.869315 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:48.869321 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:48.870095 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:48.870098 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:48.870719 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:48.870730 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:48.871259 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:48.871262 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:48.871392 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:48.871954 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:48.871966 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:48.874063 (+ 2097us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.155.126:40955: Server oryx/127.9.155.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":75,"threads_started":1}
W20260620 19:10:48.874346 19030 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:10:48.868457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.65:41109 (local address 127.9.155.126:40955)
0620 19:10:48.868672 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:48.868676 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:48.868689 (+ 13us) server_negotiation.cc:407] Connection header received
0620 19:10:48.868783 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:48.868786 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:48.868828 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:48.868909 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:48.869446 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:48.869985 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:48.870883 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:48.871138 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:48.871226 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:48.874227 (+ 3001us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.155.65:41109: BlockingRecv error: recv got EOF from 127.9.155.65:41109 (error 108)
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":65,"threads_started":1}
Jun 20 19:10:49 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.66@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:10:49 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.66@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:10:49.063413 19031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:49.057583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40955 (local address 127.9.155.66:50469)
0620 19:10:49.057989 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:49.058001 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:49.058115 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:49.058350 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:49.058352 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:49.058539 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:49.058633 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:49.058638 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:49.059525 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:49.059528 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:49.060123 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:49.060133 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:49.060676 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:49.060681 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:49.060773 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:49.061424 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:49.061446 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:49.063280 (+ 1834us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.155.126:40955: Server oryx/127.9.155.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":80,"threads_started":1}
W20260620 19:10:49.063623 19030 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:10:49.057705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.66:50469 (local address 127.9.155.126:40955)
0620 19:10:49.057845 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:49.057849 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:49.058060 (+ 211us) server_negotiation.cc:407] Connection header received
0620 19:10:49.058207 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:49.058209 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:49.058258 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:49.058324 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:49.058760 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:49.059356 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:49.060287 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:49.060555 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:49.060656 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:49.063504 (+ 2848us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.155.66:50469: BlockingRecv error: recv got EOF from 127.9.155.66:50469 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 20 19:10:49 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.67@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:10:49 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.67@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:10:49.259178 19032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:49.253776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40955 (local address 127.9.155.67:41609)
0620 19:10:49.254101 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:49.254118 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:49.254244 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:49.254511 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:49.254514 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:49.254720 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:49.254833 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:49.254840 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:49.255610 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:49.255614 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:49.256178 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:49.256187 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:49.256653 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:49.256657 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:49.256735 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:49.257230 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:49.257242 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:49.258980 (+ 1738us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.155.126:40955: Server oryx/127.9.155.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":86,"threads_started":1}
W20260620 19:10:49.259228 19030 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:10:49.253923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.67:41609 (local address 127.9.155.126:40955)
0620 19:10:49.254110 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:49.254116 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:49.254197 (+ 81us) server_negotiation.cc:407] Connection header received
0620 19:10:49.254341 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:49.254344 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:49.254399 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:49.254475 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:49.254971 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:49.255493 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:49.256328 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:49.256549 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:49.256637 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:49.259127 (+ 2490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.155.67:41609: BlockingRecv error: recv got EOF from 127.9.155.67:41609 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Jun 20 19:10:49 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.65@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:10:49 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.65@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:10:49.881052 19033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:49.875136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40955 (local address 127.9.155.65:57151)
0620 19:10:49.875474 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:49.875488 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:49.875584 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:49.875883 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:49.875886 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:49.876075 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:49.876182 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:49.876187 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:49.877081 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:49.877084 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:49.877707 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:49.877717 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:49.878217 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:49.878220 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:49.878293 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:49.878812 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:49.878827 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:49.880926 (+ 2099us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.155.126:40955: Server oryx/127.9.155.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":89,"threads_started":1}
W20260620 19:10:49.881232 19034 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:10:49.875276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.65:57151 (local address 127.9.155.126:40955)
0620 19:10:49.875628 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:49.875632 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:49.875645 (+ 13us) server_negotiation.cc:407] Connection header received
0620 19:10:49.875687 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:49.875689 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:49.875730 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:49.875816 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:49.876361 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:49.876954 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:49.877856 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:49.878087 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:49.878171 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:49.881087 (+ 2916us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.155.65:57151: BlockingRecv error: recv got EOF from 127.9.155.65:57151 (error 108)
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":75,"threads_started":1}
Jun 20 19:10:50 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.66@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:10:50 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.66@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:10:50.069347 19035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:50.064210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40955 (local address 127.9.155.66:39303)
0620 19:10:50.064487 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:50.064499 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:50.064600 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:50.064817 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:50.064819 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:50.065002 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:50.065097 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:50.065104 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:50.065825 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:50.065827 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:50.066420 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:50.066429 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:50.066837 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:50.066839 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:50.066898 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:50.067387 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:50.067397 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:50.069216 (+ 1819us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.155.126:40955: Server oryx/127.9.155.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":72,"threads_started":1}
W20260620 19:10:50.069480 19034 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:10:50.064336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.66:39303 (local address 127.9.155.126:40955)
0620 19:10:50.064474 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:50.064478 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:50.064560 (+ 82us) server_negotiation.cc:407] Connection header received
0620 19:10:50.064673 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:50.064675 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:50.064719 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:50.064795 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:50.065194 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:50.065730 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:50.066532 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:50.066751 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:50.066828 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:50.069363 (+ 2535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.155.66:39303: BlockingRecv error: recv got EOF from 127.9.155.66:39303 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 20 19:10:50 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.67@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:10:50 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.67@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:10:50.269460 19036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:50.260288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40955 (local address 127.9.155.67:52105)
0620 19:10:50.260609 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:50.260626 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:50.260751 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:50.261068 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:50.261072 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:50.261266 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:50.261368 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:50.261373 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:50.262151 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:50.262154 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:50.262760 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:50.262770 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:50.263434 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:50.263437 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:50.263515 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:50.267320 (+ 3805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:50.267335 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:50.269306 (+ 1971us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.155.126:40955: Server oryx/127.9.155.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":126,"threads_started":1}
W20260620 19:10:50.269680 19034 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:10:50.260456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.67:52105 (local address 127.9.155.126:40955)
0620 19:10:50.260610 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:50.260614 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:50.260706 (+ 92us) server_negotiation.cc:407] Connection header received
0620 19:10:50.260886 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:50.260889 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:50.260945 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:50.261014 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:50.261479 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:50.262027 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:50.262997 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:50.263233 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:50.263321 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:50.269557 (+ 6236us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.155.67:52105: BlockingRecv error: recv got EOF from 127.9.155.67:52105 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 20 19:10:50 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.65@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:10:50 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.65@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:10:50.887359 19037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:50.881908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40955 (local address 127.9.155.65:39179)
0620 19:10:50.882227 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:50.882241 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:50.882326 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:50.882556 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:50.882558 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:50.882729 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:50.882819 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:50.882824 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:50.883611 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:50.883614 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:50.884290 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:50.884305 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:50.884763 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:50.884767 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:50.884853 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:50.885335 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:50.885347 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:50.887198 (+ 1851us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.155.126:40955: Server oryx/127.9.155.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":102,"threads_started":1}
W20260620 19:10:50.887456 19038 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:10:50.882059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.65:39179 (local address 127.9.155.126:40955)
0620 19:10:50.882268 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:50.882271 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:50.882286 (+ 15us) server_negotiation.cc:407] Connection header received
0620 19:10:50.882416 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:50.882418 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:50.882460 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:50.882534 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:50.882924 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:50.883496 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:50.884410 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:50.884650 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:50.884744 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:50.887341 (+ 2597us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.155.65:39179: BlockingRecv error: recv got EOF from 127.9.155.65:39179 (error 108)
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":72,"threads_started":1}
Jun 20 19:10:51 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.66@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:10:51 dist-test-slave-n3fx krb5kdc[17980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.155.66@KRBTEST.COM for oryx/127.9.155.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:10:51.075534 19039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:51.070288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40955 (local address 127.9.155.66:48653)
0620 19:10:51.070687 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:51.070700 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:51.070789 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:51.070998 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:51.071000 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:51.071177 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:51.071270 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:51.071276 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:51.072019 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:51.072022 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:51.072646 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:51.072655 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:51.073103 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:51.073105 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:51.073166 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:51.073628 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:51.073639 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:51.075413 (+ 1774us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.155.126:40955: Server oryx/127.9.155.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":124,"threads_started":1}
W20260620 19:10:51.075726 19038 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:10:51.070417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.66:48653 (local address 127.9.155.126:40955)
0620 19:10:51.070595 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:51.070599 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:51.070761 (+ 162us) server_negotiation.cc:407] Connection header received
0620 19:10:51.070856 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:51.070858 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:51.070901 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:51.070969 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:51.071396 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:51.071917 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:51.072772 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:51.073003 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:51.073089 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:51.075589 (+ 2500us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.155.66:48653: BlockingRecv error: recv got EOF from 127.9.155.66:48653 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 19:10:51.151473 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 18563
I20260620 19:10:51.156188 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 18715
I20260620 19:10:51.160779 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 18847
I20260620 19:10:51.165258 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 18493
2026-06-20T19:10:51Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17879 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-20T19:10:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:10:51Z Disabled control of system clock
I20260620 19:10:51.192903 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982595231230-9837-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982595231230-9837-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982595231230-9837-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:40887
--webserver_interface=127.9.155.126
--webserver_port=0
--builtin_ntp_servers=127.9.155.84:32907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.155.126:40887
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 19:10:51.300629 19043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:51.300884 19043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:51.300917 19043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:51.304862 19043 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:10:51.304924 19043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:51.304939 19043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:51.304955 19043 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:10:51.304967 19043 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:10:51.309774 19043 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:32907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982595231230-9837-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982595231230-9837-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.155.126:40887
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:40887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.155.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19043
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982595231230-9837-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:10:51.310791 19043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:51.312142 19043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:10:51.320861 19048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:51.320883 19049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:51.320940 19051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:51.320942 19043 server_base.cc:1034] running on GCE node
I20260620 19:10:51.321653 19043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:51.322702 19043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:51.323874 19043 hybrid_clock.cc:648] HybridClock initialized: now 1781982651323857 us; error 33 us; skew 500 ppm
I20260620 19:10:51.325685 19043 webserver.cc:460] Webserver started at http://127.9.155.126:44697/ using document root <none> and password file <none>
I20260620 19:10:51.326253 19043 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:51.326337 19043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:51.326565 19043 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:10:51.328675 19043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982595231230-9837-0/minicluster-data/master-0/data/instance:
uuid: "b578295fb3d24f19926d620efd4f1a5a"
format_stamp: "Formatted at 2026-06-20 19:10:51 on dist-test-slave-n3fx"
server_key: "56918049d04dee33f717787e2f543114"
server_key_iv: "b899df42c1c2568922a0c4ade771e8a6"
server_key_version: "encryptionkey@0"
I20260620 19:10:51.329213 19043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982595231230-9837-0/minicluster-data/master-0/wal/instance:
uuid: "b578295fb3d24f19926d620efd4f1a5a"
format_stamp: "Formatted at 2026-06-20 19:10:51 on dist-test-slave-n3fx"
server_key: "56918049d04dee33f717787e2f543114"
server_key_iv: "b899df42c1c2568922a0c4ade771e8a6"
server_key_version: "encryptionkey@0"
I20260620 19:10:51.333027 19043 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 19:10:51.335630 19057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:51.337009 19043 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 19:10:51.337184 19043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982595231230-9837-0/minicluster-data/master-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982595231230-9837-0/minicluster-data/master-0/wal
uuid: "b578295fb3d24f19926d620efd4f1a5a"
format_stamp: "Formatted at 2026-06-20 19:10:51 on dist-test-slave-n3fx"
server_key: "56918049d04dee33f717787e2f543114"
server_key_iv: "b899df42c1c2568922a0c4ade771e8a6"
server_key_version: "encryptionkey@0"
I20260620 19:10:51.337297 19043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982595231230-9837-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982595231230-9837-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982595231230-9837-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:51.358438 19043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:51.359105 19043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:51.359253 19043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:51.365828 19043 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.126:40887
I20260620 19:10:51.365861 19109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.126:40887 every 8 connection(s)
I20260620 19:10:51.366768 19043 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
I20260620 19:10:51.368292 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 19043
I20260620 19:10:51.368381 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982595231230-9837-0/minicluster-data/master-0/wal/instance
I20260620 19:10:51.368645 9837 external_mini_cluster.cc:1428] Setting key 7cbbaa63fa67c419dd3d5254057e1b3e
I20260620 19:10:51.370558 19110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:51.376581 19110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b578295fb3d24f19926d620efd4f1a5a: Bootstrap starting.
I20260620 19:10:51.377815 19113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:51.370054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59562 (local address 127.9.155.126:40887)
0620 19:10:51.370637 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:51.370655 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:51.370701 (+ 46us) server_negotiation.cc:407] Connection header received
0620 19:10:51.371437 (+ 736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:51.371455 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:51.371865 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:51.372148 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:51.372530 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:51.373498 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:51.374625 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:51.375105 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:51.375225 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:51.376090 (+ 865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:51.376095 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:51.376108 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:10:51.376537 (+ 429us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:51.376592 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:10:51.376614 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:51.376703 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:51.376844 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 19:10:51.377109 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":104,"threads_started":1}
I20260620 19:10:51.378830 19110 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b578295fb3d24f19926d620efd4f1a5a: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:51.379540 19110 log.cc:824] T 00000000000000000000000000000000 P b578295fb3d24f19926d620efd4f1a5a: Log is configured to *not* fsync() on all Append() calls
I20260620 19:10:51.381511 19110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b578295fb3d24f19926d620efd4f1a5a: No bootstrap required, opened a new log
I20260620 19:10:51.384887 19110 raft_consensus.cc:348] T 00000000000000000000000000000000 P b578295fb3d24f19926d620efd4f1a5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b578295fb3d24f19926d620efd4f1a5a" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 40887 } }
I20260620 19:10:51.385126 19110 raft_consensus.cc:374] T 00000000000000000000000000000000 P b578295fb3d24f19926d620efd4f1a5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:51.385185 19110 raft_consensus.cc:729] T 00000000000000000000000000000000 P b578295fb3d24f19926d620efd4f1a5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b578295fb3d24f19926d620efd4f1a5a, State: Initialized, Role: FOLLOWER
I20260620 19:10:51.385589 19110 consensus_queue.cc:260] T 00000000000000000000000000000000 P b578295fb3d24f19926d620efd4f1a5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b578295fb3d24f19926d620efd4f1a5a" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 40887 } }
I20260620 19:10:51.385717 19110 raft_consensus.cc:388] T 00000000000000000000000000000000 P b578295fb3d24f19926d620efd4f1a5a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:10:51.385780 19110 raft_consensus.cc:482] T 00000000000000000000000000000000 P b578295fb3d24f19926d620efd4f1a5a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:10:51.385874 19110 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b578295fb3d24f19926d620efd4f1a5a [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:51.386785 19110 raft_consensus.cc:504] T 00000000000000000000000000000000 P b578295fb3d24f19926d620efd4f1a5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b578295fb3d24f19926d620efd4f1a5a" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 40887 } }
I20260620 19:10:51.387121 19110 leader_election.cc:302] T 00000000000000000000000000000000 P b578295fb3d24f19926d620efd4f1a5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b578295fb3d24f19926d620efd4f1a5a; no voters:
I20260620 19:10:51.387437 19110 leader_election.cc:288] T 00000000000000000000000000000000 P b578295fb3d24f19926d620efd4f1a5a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:10:51.387539 19115 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b578295fb3d24f19926d620efd4f1a5a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:10:51.387758 19115 raft_consensus.cc:686] T 00000000000000000000000000000000 P b578295fb3d24f19926d620efd4f1a5a [term 1 LEADER]: Becoming Leader. State: Replica: b578295fb3d24f19926d620efd4f1a5a, State: Running, Role: LEADER
I20260620 19:10:51.388128 19115 consensus_queue.cc:237] T 00000000000000000000000000000000 P b578295fb3d24f19926d620efd4f1a5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b578295fb3d24f19926d620efd4f1a5a" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 40887 } }
I20260620 19:10:51.388691 19110 sys_catalog.cc:564] T 00000000000000000000000000000000 P b578295fb3d24f19926d620efd4f1a5a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:10:51.389837 19116 sys_catalog.cc:455] T 00000000000000000000000000000000 P b578295fb3d24f19926d620efd4f1a5a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b578295fb3d24f19926d620efd4f1a5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b578295fb3d24f19926d620efd4f1a5a" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 40887 } } }
I20260620 19:10:51.389988 19116 sys_catalog.cc:458] T 00000000000000000000000000000000 P b578295fb3d24f19926d620efd4f1a5a [sys.catalog]: This master's current role is: LEADER
I20260620 19:10:51.389837 19117 sys_catalog.cc:455] T 00000000000000000000000000000000 P b578295fb3d24f19926d620efd4f1a5a [sys.catalog]: SysCatalogTable state changed. Reason: New leader b578295fb3d24f19926d620efd4f1a5a. Latest consensus state: current_term: 1 leader_uuid: "b578295fb3d24f19926d620efd4f1a5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b578295fb3d24f19926d620efd4f1a5a" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 40887 } } }
I20260620 19:10:51.390147 19117 sys_catalog.cc:458] T 00000000000000000000000000000000 P b578295fb3d24f19926d620efd4f1a5a [sys.catalog]: This master's current role is: LEADER
W20260620 19:10:51.393842 19130 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b578295fb3d24f19926d620efd4f1a5a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:10:51.393954 19130 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:10:51.394052 19124 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:10:51.394798 19124 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:10:51.400100 19124 catalog_manager.cc:1347] Generated new cluster ID: f72f0a7b796141f2b0b5665d65fe20ce
I20260620 19:10:51.400178 19124 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:10:51.414631 19124 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:10:51.415935 19124 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:10:51.433473 19124 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b578295fb3d24f19926d620efd4f1a5a: Generated new TSK 0
I20260620 19:10:51.434302 19124 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:10:51.447954 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.65:0
--local_ip_for_outbound_sockets=127.9.155.65
--webserver_interface=127.9.155.65
--webserver_port=0
--tserver_master_addrs=127.9.155.126:40887
--builtin_ntp_servers=127.9.155.84:32907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 19:10:51.561710 19134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:51.561993 19134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:51.562029 19134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:51.566179 19134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:51.566246 19134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:51.566332 19134 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.65
I20260620 19:10:51.571286 19134 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:32907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.155.65
--webserver_port=0
--tserver_master_addrs=127.9.155.126:40887
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19134
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.65
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:10:51.572484 19134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:51.573927 19134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:10:51.583303 19139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:51.583328 19142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:51.583521 19134 server_base.cc:1034] running on GCE node
W20260620 19:10:51.583285 19140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:51.583986 19134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:51.584718 19134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:51.585927 19134 hybrid_clock.cc:648] HybridClock initialized: now 1781982651585911 us; error 34 us; skew 500 ppm
I20260620 19:10:51.587881 19134 webserver.cc:460] Webserver started at http://127.9.155.65:41053/ using document root <none> and password file <none>
I20260620 19:10:51.588506 19134 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:51.588570 19134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:51.588804 19134 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:10:51.591012 19134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-0/data/instance:
uuid: "0d971ed47c824115bd36c43f9736ecdb"
format_stamp: "Formatted at 2026-06-20 19:10:51 on dist-test-slave-n3fx"
server_key: "fb8b4f25287e4defe948690e5af3c8d5"
server_key_iv: "08a6c265e3dfdb652ba46d8e0743d41b"
server_key_version: "encryptionkey@0"
I20260620 19:10:51.591543 19134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-0/wal/instance:
uuid: "0d971ed47c824115bd36c43f9736ecdb"
format_stamp: "Formatted at 2026-06-20 19:10:51 on dist-test-slave-n3fx"
server_key: "fb8b4f25287e4defe948690e5af3c8d5"
server_key_iv: "08a6c265e3dfdb652ba46d8e0743d41b"
server_key_version: "encryptionkey@0"
I20260620 19:10:51.595257 19134 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 19:10:51.598091 19148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:51.599514 19134 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 19:10:51.599627 19134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-0/wal
uuid: "0d971ed47c824115bd36c43f9736ecdb"
format_stamp: "Formatted at 2026-06-20 19:10:51 on dist-test-slave-n3fx"
server_key: "fb8b4f25287e4defe948690e5af3c8d5"
server_key_iv: "08a6c265e3dfdb652ba46d8e0743d41b"
server_key_version: "encryptionkey@0"
I20260620 19:10:51.599761 19134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:51.639496 19134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:51.640246 19134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:51.640426 19134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:51.641065 19134 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:10:51.642015 19134 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:10:51.642061 19134 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:51.642102 19134 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:10:51.642130 19134 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:51.650166 19134 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.65:35971
I20260620 19:10:51.650187 19261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.65:35971 every 8 connection(s)
I20260620 19:10:51.651034 19134 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
I20260620 19:10:51.654508 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 19134
I20260620 19:10:51.654609 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-0/wal/instance
I20260620 19:10:51.654871 9837 external_mini_cluster.cc:1428] Setting key d1a1650f025467c5c362432470d9e2ff
I20260620 19:10:51.656983 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.66:0
--local_ip_for_outbound_sockets=127.9.155.66
--webserver_interface=127.9.155.66
--webserver_port=0
--tserver_master_addrs=127.9.155.126:40887
--builtin_ntp_servers=127.9.155.84:32907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 19:10:51.661036 19113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:51.652779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.65:56521 (local address 127.9.155.126:40887)
0620 19:10:51.652920 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:51.652923 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:51.653979 (+ 1056us) server_negotiation.cc:407] Connection header received
0620 19:10:51.655407 (+ 1428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:51.655411 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:51.655469 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:51.655561 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:51.656686 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:51.657458 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:51.658934 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:51.659161 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:51.659249 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:51.660533 (+ 1284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:51.660537 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:51.660540 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:10:51.660558 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:51.660581 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:10:51.660590 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:51.660651 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:51.660820 (+ 169us) server_negotiation.cc:299] Negotiation successful
0620 19:10:51.660884 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 19:10:51.662168 19264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:51.653184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40887 (local address 127.9.155.65:56521)
0620 19:10:51.653836 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:51.653875 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:51.655174 (+ 1299us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:51.655740 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:51.655748 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:51.655850 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:51.656531 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:51.656548 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:51.657607 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:51.657610 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:51.658781 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:51.658793 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:51.659323 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:51.659328 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:51.659564 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:51.660179 (+ 615us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:10:51.660181 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:51.660264 (+ 83us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:10:51.660351 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:51.660673 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:51.660682 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:51.660689 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 19:10:51.660824 (+ 135us) client_negotiation.cc:240] Negotiation successful
0620 19:10:51.661139 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":110,"threads_started":1}
I20260620 19:10:51.663372 19262 heartbeater.cc:347] Connected to a master server at 127.9.155.126:40887
I20260620 19:10:51.663638 19262 heartbeater.cc:464] Registering TS with master...
I20260620 19:10:51.664304 19262 heartbeater.cc:511] Master 127.9.155.126:40887 requested a full tablet report, sending...
I20260620 19:10:51.665829 19074 ts_manager.cc:194] Registered new tserver with Master: 0d971ed47c824115bd36c43f9736ecdb (127.9.155.65:35971)
I20260620 19:10:51.666955 19074 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.155.65:56521
W20260620 19:10:51.767271 19265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:51.767547 19265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:51.767599 19265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:51.771744 19265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:51.771819 19265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:51.771906 19265 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.66
I20260620 19:10:51.776878 19265 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:32907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.155.66
--webserver_port=0
--tserver_master_addrs=127.9.155.126:40887
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19265
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.66
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:10:51.777985 19265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:51.779318 19265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 19:10:51.788655 19265 server_base.cc:1034] running on GCE node
W20260620 19:10:51.788609 19270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:51.788609 19273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:51.788822 19271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:51.789278 19265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:51.789974 19265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:51.791158 19265 hybrid_clock.cc:648] HybridClock initialized: now 1781982651791134 us; error 40 us; skew 500 ppm
I20260620 19:10:51.793205 19265 webserver.cc:460] Webserver started at http://127.9.155.66:42467/ using document root <none> and password file <none>
I20260620 19:10:51.793812 19265 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:51.793877 19265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:51.794090 19265 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:10:51.796190 19265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-1/data/instance:
uuid: "8477a2440c52453eb9be93a70217843e"
format_stamp: "Formatted at 2026-06-20 19:10:51 on dist-test-slave-n3fx"
server_key: "b084214ce13e20de74a458098976e462"
server_key_iv: "4977bcd0c4cf0a179ab8a5a78ed51af3"
server_key_version: "encryptionkey@0"
I20260620 19:10:51.796783 19265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-1/wal/instance:
uuid: "8477a2440c52453eb9be93a70217843e"
format_stamp: "Formatted at 2026-06-20 19:10:51 on dist-test-slave-n3fx"
server_key: "b084214ce13e20de74a458098976e462"
server_key_iv: "4977bcd0c4cf0a179ab8a5a78ed51af3"
server_key_version: "encryptionkey@0"
I20260620 19:10:51.800613 19265 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 19:10:51.803216 19279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:51.804575 19265 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:10:51.804672 19265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-1/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-1/wal
uuid: "8477a2440c52453eb9be93a70217843e"
format_stamp: "Formatted at 2026-06-20 19:10:51 on dist-test-slave-n3fx"
server_key: "b084214ce13e20de74a458098976e462"
server_key_iv: "4977bcd0c4cf0a179ab8a5a78ed51af3"
server_key_version: "encryptionkey@0"
I20260620 19:10:51.804767 19265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:51.814913 19265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:51.815559 19265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:51.815711 19265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:51.816254 19265 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:10:51.817173 19265 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:10:51.817219 19265 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:51.817260 19265 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:10:51.817273 19265 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:51.825599 19265 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.66:38769
I20260620 19:10:51.825655 19392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.66:38769 every 8 connection(s)
I20260620 19:10:51.826570 19265 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
I20260620 19:10:51.833491 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 19265
I20260620 19:10:51.833639 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-1/wal/instance
I20260620 19:10:51.833910 9837 external_mini_cluster.cc:1428] Setting key 9aae0b66cb140af45e8e7223a35cce48
I20260620 19:10:51.836064 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.67:0
--local_ip_for_outbound_sockets=127.9.155.67
--webserver_interface=127.9.155.67
--webserver_port=0
--tserver_master_addrs=127.9.155.126:40887
--builtin_ntp_servers=127.9.155.84:32907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 19:10:51.836373 19113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:51.828427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.66:41977 (local address 127.9.155.126:40887)
0620 19:10:51.828626 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:51.828631 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:51.829774 (+ 1143us) server_negotiation.cc:407] Connection header received
0620 19:10:51.831167 (+ 1393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:51.831170 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:51.831223 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:51.831305 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:51.832327 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:51.832921 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:51.834347 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:51.834582 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:51.834697 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:51.835850 (+ 1153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:51.835854 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:51.835856 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:10:51.835868 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:51.835887 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:10:51.835895 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:51.835951 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:51.836122 (+ 171us) server_negotiation.cc:299] Negotiation successful
0620 19:10:51.836193 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260620 19:10:51.837527 19395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:51.828846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40887 (local address 127.9.155.66:41977)
0620 19:10:51.829618 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:51.829663 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:51.830915 (+ 1252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:51.831518 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:51.831525 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:51.831616 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:51.832136 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:51.832149 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:51.833070 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:51.833074 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:51.834211 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:51.834223 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:51.834748 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:51.834751 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:51.834911 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:51.835543 (+ 632us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:10:51.835545 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:51.835603 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:10:51.835693 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:51.835976 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:51.835986 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:51.835993 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 19:10:51.836132 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 19:10:51.836475 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":138,"threads_started":1}
I20260620 19:10:51.838750 19393 heartbeater.cc:347] Connected to a master server at 127.9.155.126:40887
I20260620 19:10:51.838992 19393 heartbeater.cc:464] Registering TS with master...
I20260620 19:10:51.839507 19393 heartbeater.cc:511] Master 127.9.155.126:40887 requested a full tablet report, sending...
I20260620 19:10:51.840557 19074 ts_manager.cc:194] Registered new tserver with Master: 8477a2440c52453eb9be93a70217843e (127.9.155.66:38769)
I20260620 19:10:51.841123 19074 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.155.66:41977
W20260620 19:10:51.947394 19396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:51.947669 19396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:51.947721 19396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:51.951807 19396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:51.951869 19396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:51.951942 19396 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.67
I20260620 19:10:51.956812 19396 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:32907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.155.67
--webserver_port=0
--tserver_master_addrs=127.9.155.126:40887
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19396
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.67
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:10:51.957948 19396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:51.959391 19396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:10:51.968966 19402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:51.968969 19401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:51.968969 19404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:51.969257 19396 server_base.cc:1034] running on GCE node
I20260620 19:10:51.969682 19396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:51.970304 19396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:51.971504 19396 hybrid_clock.cc:648] HybridClock initialized: now 1781982651971482 us; error 35 us; skew 500 ppm
I20260620 19:10:51.973474 19396 webserver.cc:460] Webserver started at http://127.9.155.67:45277/ using document root <none> and password file <none>
I20260620 19:10:51.974030 19396 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:51.974092 19396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:51.974291 19396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:10:51.976428 19396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-2/data/instance:
uuid: "305782019f67424e95e65dd4fd9c3b11"
format_stamp: "Formatted at 2026-06-20 19:10:51 on dist-test-slave-n3fx"
server_key: "3f56c9f1d90f1487b1c00b8e140e9376"
server_key_iv: "0ce3fb37c934ce7399d9dfcd83d0f0e8"
server_key_version: "encryptionkey@0"
I20260620 19:10:51.976964 19396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-2/wal/instance:
uuid: "305782019f67424e95e65dd4fd9c3b11"
format_stamp: "Formatted at 2026-06-20 19:10:51 on dist-test-slave-n3fx"
server_key: "3f56c9f1d90f1487b1c00b8e140e9376"
server_key_iv: "0ce3fb37c934ce7399d9dfcd83d0f0e8"
server_key_version: "encryptionkey@0"
I20260620 19:10:51.980512 19396 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 19:10:51.983062 19410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:51.984310 19396 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:10:51.984453 19396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-2/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-2/wal
uuid: "305782019f67424e95e65dd4fd9c3b11"
format_stamp: "Formatted at 2026-06-20 19:10:51 on dist-test-slave-n3fx"
server_key: "3f56c9f1d90f1487b1c00b8e140e9376"
server_key_iv: "0ce3fb37c934ce7399d9dfcd83d0f0e8"
server_key_version: "encryptionkey@0"
I20260620 19:10:51.984556 19396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:51.994131 19396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:51.994853 19396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:51.995033 19396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:51.995637 19396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:10:51.996637 19396 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:10:51.996690 19396 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:51.996734 19396 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:10:51.996762 19396 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:52.004884 19396 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.67:43815
I20260620 19:10:52.004947 19523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.67:43815 every 8 connection(s)
I20260620 19:10:52.005780 19396 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
I20260620 19:10:52.012573 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 19396
I20260620 19:10:52.012655 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-2/wal/instance
I20260620 19:10:52.012872 9837 external_mini_cluster.cc:1428] Setting key 157ce3dbf3253ead9bea21a43e24b95c
I20260620 19:10:52.014922 19113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:52.007547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.67:41231 (local address 127.9.155.126:40887)
0620 19:10:52.007690 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:52.007693 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:52.008806 (+ 1113us) server_negotiation.cc:407] Connection header received
0620 19:10:52.010178 (+ 1372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:52.010182 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:52.010238 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:52.010311 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:52.011205 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:52.011745 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:52.012986 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:52.013238 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:52.013363 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:52.014477 (+ 1114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:52.014481 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:52.014484 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:10:52.014496 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:52.014519 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:10:52.014529 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:52.014595 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:52.014719 (+ 124us) server_negotiation.cc:299] Negotiation successful
0620 19:10:52.014778 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 19:10:52.015766 19526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:52.008004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40887 (local address 127.9.155.67:41231)
0620 19:10:52.008668 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:52.008708 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:52.009983 (+ 1275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:52.010487 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:52.010493 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:52.010575 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:52.011073 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:52.011087 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:52.011870 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:52.011873 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:52.012844 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:52.012854 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:52.013416 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:52.013421 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:52.013611 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:52.014192 (+ 581us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:10:52.014195 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:52.014250 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:10:52.014333 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:52.014606 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:52.014610 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:52.014614 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:10:52.014699 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 19:10:52.014987 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":140,"threads_started":1}
I20260620 19:10:52.016789 19524 heartbeater.cc:347] Connected to a master server at 127.9.155.126:40887
I20260620 19:10:52.017001 19524 heartbeater.cc:464] Registering TS with master...
I20260620 19:10:52.017480 19524 heartbeater.cc:511] Master 127.9.155.126:40887 requested a full tablet report, sending...
I20260620 19:10:52.018468 19074 ts_manager.cc:194] Registered new tserver with Master: 305782019f67424e95e65dd4fd9c3b11 (127.9.155.67:43815)
I20260620 19:10:52.019075 19074 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.155.67:41231
I20260620 19:10:52.027061 9837 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 19:10:52.029798 19113 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:10:52.028940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59570 (local address 127.9.155.126:40887)
0620 19:10:52.029060 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:52.029063 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:52.029227 (+ 164us) server_negotiation.cc:407] Connection header received
0620 19:10:52.029331 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:52.029333 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:52.029376 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:52.029482 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:52.029662 (+ 180us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59570: BlockingRecv error: recv got EOF from 127.0.0.1:59570 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 19:10:52.030077 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 19134
I20260620 19:10:52.035583 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 19265
I20260620 19:10:52.040597 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 19396
I20260620 19:10:52.045563 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 19043
2026-06-20T19:10:52Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (879 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-20T19:10:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:10:52Z Disabled control of system clock
I20260620 19:10:52.073343 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982595231230-9837-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982595231230-9837-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982595231230-9837-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:41791
--webserver_interface=127.9.155.126
--webserver_port=0
--builtin_ntp_servers=127.9.155.84:37831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.155.126:41791
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 19:10:52.182139 19535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:52.182435 19535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:52.182473 19535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:52.186569 19535 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:10:52.186650 19535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:52.186664 19535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:52.186681 19535 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:10:52.186694 19535 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:10:52.192094 19535 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:37831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982595231230-9837-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982595231230-9837-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.155.126:41791
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:41791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.155.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19535
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982595231230-9837-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:10:52.193244 19535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:52.194638 19535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:10:52.202970 19540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:52.203032 19535 server_base.cc:1034] running on GCE node
W20260620 19:10:52.202979 19541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:52.202957 19543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:52.203805 19535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:52.204960 19535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:52.206133 19535 hybrid_clock.cc:648] HybridClock initialized: now 1781982652206119 us; error 33 us; skew 500 ppm
I20260620 19:10:52.208040 19535 webserver.cc:460] Webserver started at http://127.9.155.126:34155/ using document root <none> and password file <none>
I20260620 19:10:52.208648 19535 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:52.208714 19535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:52.208879 19535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:10:52.210896 19535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982595231230-9837-0/minicluster-data/master-0/data/instance:
uuid: "1f7502d8bded4c54ac1ec1332fda0bb6"
format_stamp: "Formatted at 2026-06-20 19:10:52 on dist-test-slave-n3fx"
server_key: "38dd4f95eae1404bf224a1f8bff5312c"
server_key_iv: "0964ffce628eb24674a628c0f36f6bcc"
server_key_version: "encryptionkey@0"
I20260620 19:10:52.211392 19535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982595231230-9837-0/minicluster-data/master-0/wal/instance:
uuid: "1f7502d8bded4c54ac1ec1332fda0bb6"
format_stamp: "Formatted at 2026-06-20 19:10:52 on dist-test-slave-n3fx"
server_key: "38dd4f95eae1404bf224a1f8bff5312c"
server_key_iv: "0964ffce628eb24674a628c0f36f6bcc"
server_key_version: "encryptionkey@0"
I20260620 19:10:52.215036 19535 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 19:10:52.217370 19549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:52.218623 19535 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 19:10:52.218725 19535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982595231230-9837-0/minicluster-data/master-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982595231230-9837-0/minicluster-data/master-0/wal
uuid: "1f7502d8bded4c54ac1ec1332fda0bb6"
format_stamp: "Formatted at 2026-06-20 19:10:52 on dist-test-slave-n3fx"
server_key: "38dd4f95eae1404bf224a1f8bff5312c"
server_key_iv: "0964ffce628eb24674a628c0f36f6bcc"
server_key_version: "encryptionkey@0"
I20260620 19:10:52.218834 19535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982595231230-9837-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982595231230-9837-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982595231230-9837-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:52.223376 19535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:52.224020 19535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:52.224174 19535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:52.230616 19535 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.126:41791
I20260620 19:10:52.230638 19601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.126:41791 every 8 connection(s)
I20260620 19:10:52.231554 19535 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
I20260620 19:10:52.234671 19602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:52.238817 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 19535
I20260620 19:10:52.238922 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982595231230-9837-0/minicluster-data/master-0/wal/instance
I20260620 19:10:52.239184 9837 external_mini_cluster.cc:1428] Setting key 12f765bfc0cb6a61d80e8bd295df1b06
I20260620 19:10:52.241328 19602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f7502d8bded4c54ac1ec1332fda0bb6: Bootstrap starting.
I20260620 19:10:52.243778 19602 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f7502d8bded4c54ac1ec1332fda0bb6: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:52.244639 19602 log.cc:824] T 00000000000000000000000000000000 P 1f7502d8bded4c54ac1ec1332fda0bb6: Log is configured to *not* fsync() on all Append() calls
I20260620 19:10:52.245078 19605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:52.240656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34598 (local address 127.9.155.126:41791)
0620 19:10:52.241333 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:52.241349 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:52.241386 (+ 37us) server_negotiation.cc:407] Connection header received
0620 19:10:52.242075 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:52.242094 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:52.242509 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:52.242822 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:52.243661 (+ 839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:52.243666 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:52.243678 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:10:52.243690 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:52.243746 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:10:52.243762 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:52.243862 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:52.243993 (+ 131us) server_negotiation.cc:299] Negotiation successful
0620 19:10:52.244275 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":113,"threads_started":1}
I20260620 19:10:52.246840 19602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f7502d8bded4c54ac1ec1332fda0bb6: No bootstrap required, opened a new log
I20260620 19:10:52.250489 19602 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f7502d8bded4c54ac1ec1332fda0bb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f7502d8bded4c54ac1ec1332fda0bb6" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 41791 } }
I20260620 19:10:52.250725 19602 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1f7502d8bded4c54ac1ec1332fda0bb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:52.250782 19602 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f7502d8bded4c54ac1ec1332fda0bb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f7502d8bded4c54ac1ec1332fda0bb6, State: Initialized, Role: FOLLOWER
I20260620 19:10:52.251179 19602 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f7502d8bded4c54ac1ec1332fda0bb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f7502d8bded4c54ac1ec1332fda0bb6" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 41791 } }
I20260620 19:10:52.251308 19602 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1f7502d8bded4c54ac1ec1332fda0bb6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:10:52.251372 19602 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f7502d8bded4c54ac1ec1332fda0bb6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:10:52.251475 19602 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f7502d8bded4c54ac1ec1332fda0bb6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:52.252419 19602 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f7502d8bded4c54ac1ec1332fda0bb6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f7502d8bded4c54ac1ec1332fda0bb6" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 41791 } }
I20260620 19:10:52.252738 19602 leader_election.cc:302] T 00000000000000000000000000000000 P 1f7502d8bded4c54ac1ec1332fda0bb6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1f7502d8bded4c54ac1ec1332fda0bb6; no voters:
I20260620 19:10:52.253034 19602 leader_election.cc:288] T 00000000000000000000000000000000 P 1f7502d8bded4c54ac1ec1332fda0bb6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:10:52.253134 19607 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1f7502d8bded4c54ac1ec1332fda0bb6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:10:52.253338 19607 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1f7502d8bded4c54ac1ec1332fda0bb6 [term 1 LEADER]: Becoming Leader. State: Replica: 1f7502d8bded4c54ac1ec1332fda0bb6, State: Running, Role: LEADER
I20260620 19:10:52.253685 19607 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f7502d8bded4c54ac1ec1332fda0bb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f7502d8bded4c54ac1ec1332fda0bb6" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 41791 } }
I20260620 19:10:52.254196 19602 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f7502d8bded4c54ac1ec1332fda0bb6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:10:52.255407 19609 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f7502d8bded4c54ac1ec1332fda0bb6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f7502d8bded4c54ac1ec1332fda0bb6. Latest consensus state: current_term: 1 leader_uuid: "1f7502d8bded4c54ac1ec1332fda0bb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f7502d8bded4c54ac1ec1332fda0bb6" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 41791 } } }
I20260620 19:10:52.255411 19608 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f7502d8bded4c54ac1ec1332fda0bb6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1f7502d8bded4c54ac1ec1332fda0bb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f7502d8bded4c54ac1ec1332fda0bb6" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 41791 } } }
I20260620 19:10:52.255540 19608 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f7502d8bded4c54ac1ec1332fda0bb6 [sys.catalog]: This master's current role is: LEADER
I20260620 19:10:52.255541 19609 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f7502d8bded4c54ac1ec1332fda0bb6 [sys.catalog]: This master's current role is: LEADER
W20260620 19:10:52.258709 19622 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1f7502d8bded4c54ac1ec1332fda0bb6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:10:52.258790 19622 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:10:52.258888 19617 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:10:52.259939 19617 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:10:52.265694 19617 catalog_manager.cc:1347] Generated new cluster ID: ec2dddbccc964b9887673044ef0411d5
I20260620 19:10:52.265782 19617 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:10:52.297483 19617 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:10:52.297834 19617 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:10:52.306996 19617 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1f7502d8bded4c54ac1ec1332fda0bb6: Generated new TSK 0
I20260620 19:10:52.307739 19617 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:10:52.320627 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.65:0
--local_ip_for_outbound_sockets=127.9.155.65
--webserver_interface=127.9.155.65
--webserver_port=0
--tserver_master_addrs=127.9.155.126:41791
--builtin_ntp_servers=127.9.155.84:37831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 19:10:52.438210 19626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:52.438498 19626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:52.438544 19626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:52.442632 19626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:52.442688 19626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:52.442744 19626 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.65
I20260620 19:10:52.447403 19626 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:37831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.155.65
--webserver_port=0
--tserver_master_addrs=127.9.155.126:41791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19626
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.65
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:10:52.448443 19626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:52.449743 19626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 19:10:52.459223 19626 server_base.cc:1034] running on GCE node
W20260620 19:10:52.459170 19631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:52.459172 19632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:52.459173 19634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:52.459818 19626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:52.460471 19626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:52.461640 19626 hybrid_clock.cc:648] HybridClock initialized: now 1781982652461616 us; error 37 us; skew 500 ppm
I20260620 19:10:52.463547 19626 webserver.cc:460] Webserver started at http://127.9.155.65:34261/ using document root <none> and password file <none>
I20260620 19:10:52.464114 19626 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:52.464178 19626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:52.464406 19626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:10:52.466478 19626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-0/data/instance:
uuid: "ca085ea86bb848b39e7f0151e6365a3d"
format_stamp: "Formatted at 2026-06-20 19:10:52 on dist-test-slave-n3fx"
server_key: "717fb69ad376dff0a598a393f6f7cfb6"
server_key_iv: "8c6248537e2745b253b7a32d33e4a87e"
server_key_version: "encryptionkey@0"
I20260620 19:10:52.467042 19626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-0/wal/instance:
uuid: "ca085ea86bb848b39e7f0151e6365a3d"
format_stamp: "Formatted at 2026-06-20 19:10:52 on dist-test-slave-n3fx"
server_key: "717fb69ad376dff0a598a393f6f7cfb6"
server_key_iv: "8c6248537e2745b253b7a32d33e4a87e"
server_key_version: "encryptionkey@0"
I20260620 19:10:52.470873 19626 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 19:10:52.473657 19640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:52.475055 19626 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 19:10:52.475189 19626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-0/wal
uuid: "ca085ea86bb848b39e7f0151e6365a3d"
format_stamp: "Formatted at 2026-06-20 19:10:52 on dist-test-slave-n3fx"
server_key: "717fb69ad376dff0a598a393f6f7cfb6"
server_key_iv: "8c6248537e2745b253b7a32d33e4a87e"
server_key_version: "encryptionkey@0"
I20260620 19:10:52.475404 19626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:52.480203 19626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:52.480882 19626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:52.481032 19626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:52.481546 19626 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:10:52.482442 19626 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:10:52.482486 19626 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:52.482525 19626 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:10:52.482542 19626 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:52.490921 19626 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.65:34951
I20260620 19:10:52.491070 19753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.65:34951 every 8 connection(s)
I20260620 19:10:52.491854 19626 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
I20260620 19:10:52.498225 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 19626
I20260620 19:10:52.498356 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-0/wal/instance
I20260620 19:10:52.498670 19605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:52.493879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.65:55165 (local address 127.9.155.126:41791)
0620 19:10:52.494061 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:52.494065 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:52.495132 (+ 1067us) server_negotiation.cc:407] Connection header received
0620 19:10:52.496575 (+ 1443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:52.496578 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:52.496622 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:52.496713 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:52.498118 (+ 1405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:52.498122 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:52.498124 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:10:52.498141 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:52.498168 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:10:52.498178 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:52.498262 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:52.498486 (+ 224us) server_negotiation.cc:299] Negotiation successful
0620 19:10:52.498549 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260620 19:10:52.498694 9837 external_mini_cluster.cc:1428] Setting key 5b559cb0f95cf5da8fb289b9dcdde59c
I20260620 19:10:52.499914 19756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:52.494259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:41791 (local address 127.9.155.65:55165)
0620 19:10:52.494971 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:52.495015 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:52.496326 (+ 1311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:52.496941 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:52.496949 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:52.497046 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:52.497675 (+ 629us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:10:52.497676 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:52.497722 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:10:52.497799 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:52.498255 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:52.498266 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:52.498272 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 19:10:52.498546 (+ 274us) client_negotiation.cc:240] Negotiation successful
0620 19:10:52.498899 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":127,"threads_started":1}
I20260620 19:10:52.500916 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.66:0
--local_ip_for_outbound_sockets=127.9.155.66
--webserver_interface=127.9.155.66
--webserver_port=0
--tserver_master_addrs=127.9.155.126:41791
--builtin_ntp_servers=127.9.155.84:37831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 19:10:52.501154 19754 heartbeater.cc:347] Connected to a master server at 127.9.155.126:41791
I20260620 19:10:52.501396 19754 heartbeater.cc:464] Registering TS with master...
I20260620 19:10:52.501806 19754 heartbeater.cc:511] Master 127.9.155.126:41791 requested a full tablet report, sending...
I20260620 19:10:52.503597 19566 ts_manager.cc:194] Registered new tserver with Master: ca085ea86bb848b39e7f0151e6365a3d (127.9.155.65:34951)
W20260620 19:10:52.614388 19757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:52.614663 19757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:52.614702 19757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:52.618863 19757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:52.618939 19757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:52.618999 19757 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.66
I20260620 19:10:52.624101 19757 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:37831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.155.66
--webserver_port=0
--tserver_master_addrs=127.9.155.126:41791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.66
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:10:52.625226 19757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:52.626719 19757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:10:52.636497 19762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:52.636620 19757 server_base.cc:1034] running on GCE node
W20260620 19:10:52.636427 19765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:52.636444 19763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:52.637183 19757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:52.637888 19757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:52.639066 19757 hybrid_clock.cc:648] HybridClock initialized: now 1781982652639047 us; error 33 us; skew 500 ppm
I20260620 19:10:52.641100 19757 webserver.cc:460] Webserver started at http://127.9.155.66:43845/ using document root <none> and password file <none>
I20260620 19:10:52.641680 19757 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:52.641752 19757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:52.641968 19757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:10:52.644057 19757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-1/data/instance:
uuid: "53de6949f44d4f7b8fedc8c1403f080c"
format_stamp: "Formatted at 2026-06-20 19:10:52 on dist-test-slave-n3fx"
server_key: "ffa2028f75e146870697136caff1ce92"
server_key_iv: "1a1938d234255107490c6293b1c45e2b"
server_key_version: "encryptionkey@0"
I20260620 19:10:52.644672 19757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-1/wal/instance:
uuid: "53de6949f44d4f7b8fedc8c1403f080c"
format_stamp: "Formatted at 2026-06-20 19:10:52 on dist-test-slave-n3fx"
server_key: "ffa2028f75e146870697136caff1ce92"
server_key_iv: "1a1938d234255107490c6293b1c45e2b"
server_key_version: "encryptionkey@0"
I20260620 19:10:52.648597 19757 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 19:10:52.651418 19771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:52.652801 19757 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:10:52.652963 19757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-1/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-1/wal
uuid: "53de6949f44d4f7b8fedc8c1403f080c"
format_stamp: "Formatted at 2026-06-20 19:10:52 on dist-test-slave-n3fx"
server_key: "ffa2028f75e146870697136caff1ce92"
server_key_iv: "1a1938d234255107490c6293b1c45e2b"
server_key_version: "encryptionkey@0"
I20260620 19:10:52.653085 19757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:52.658023 19757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:52.658667 19757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:52.658833 19757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:52.659385 19757 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:10:52.660369 19757 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:10:52.660420 19757 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:52.660471 19757 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:10:52.660499 19757 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:52.669152 19757 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.66:44003
I20260620 19:10:52.669179 19884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.66:44003 every 8 connection(s)
I20260620 19:10:52.670047 19757 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
I20260620 19:10:52.676064 19605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:52.671870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.66:56103 (local address 127.9.155.126:41791)
0620 19:10:52.672033 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:52.672037 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:52.673166 (+ 1129us) server_negotiation.cc:407] Connection header received
0620 19:10:52.674454 (+ 1288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:52.674458 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:52.674502 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:52.674566 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:52.675608 (+ 1042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:52.675611 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:52.675614 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:10:52.675627 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:52.675647 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:10:52.675655 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:52.675733 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:52.675860 (+ 127us) server_negotiation.cc:299] Negotiation successful
0620 19:10:52.675931 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 19:10:52.676908 19887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:52.672297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:41791 (local address 127.9.155.66:56103)
0620 19:10:52.673020 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:52.673060 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:52.674242 (+ 1182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:52.674726 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:52.674735 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:52.674820 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:52.675318 (+ 498us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:10:52.675320 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:52.675356 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:10:52.675463 (+ 107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:52.675722 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:52.675729 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:52.675735 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 19:10:52.675905 (+ 170us) client_negotiation.cc:240] Negotiation successful
0620 19:10:52.676136 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":114,"threads_started":1}
I20260620 19:10:52.677357 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 19757
I20260620 19:10:52.677480 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-1/wal/instance
I20260620 19:10:52.677780 9837 external_mini_cluster.cc:1428] Setting key d58828a55fcb6cad2cbd394685dbe4b8
I20260620 19:10:52.678100 19885 heartbeater.cc:347] Connected to a master server at 127.9.155.126:41791
I20260620 19:10:52.678318 19885 heartbeater.cc:464] Registering TS with master...
I20260620 19:10:52.678808 19885 heartbeater.cc:511] Master 127.9.155.126:41791 requested a full tablet report, sending...
I20260620 19:10:52.679692 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.67:0
--local_ip_for_outbound_sockets=127.9.155.67
--webserver_interface=127.9.155.67
--webserver_port=0
--tserver_master_addrs=127.9.155.126:41791
--builtin_ntp_servers=127.9.155.84:37831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 19:10:52.679840 19566 ts_manager.cc:194] Registered new tserver with Master: 53de6949f44d4f7b8fedc8c1403f080c (127.9.155.66:44003)
W20260620 19:10:52.787879 19888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:52.788132 19888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:52.788172 19888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:52.792268 19888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:52.792341 19888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:52.792397 19888 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.67
I20260620 19:10:52.797169 19888 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:37831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.155.67
--webserver_port=0
--tserver_master_addrs=127.9.155.126:41791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19888
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.67
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:10:52.798177 19888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:52.799463 19888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:10:52.808775 19896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:52.808815 19888 server_base.cc:1034] running on GCE node
W20260620 19:10:52.808784 19893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:52.809048 19894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:52.809407 19888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:52.810078 19888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:52.811231 19888 hybrid_clock.cc:648] HybridClock initialized: now 1781982652811214 us; error 31 us; skew 500 ppm
I20260620 19:10:52.813004 19888 webserver.cc:460] Webserver started at http://127.9.155.67:38317/ using document root <none> and password file <none>
I20260620 19:10:52.813601 19888 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:52.813668 19888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:52.813861 19888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:10:52.815850 19888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-2/data/instance:
uuid: "449ad3709aca4a27b7012df6bbd7cc2c"
format_stamp: "Formatted at 2026-06-20 19:10:52 on dist-test-slave-n3fx"
server_key: "591790b5684535df6290e88f470e684a"
server_key_iv: "0202de4dcd15ae657e522a0ddc3f4df1"
server_key_version: "encryptionkey@0"
I20260620 19:10:52.816403 19888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-2/wal/instance:
uuid: "449ad3709aca4a27b7012df6bbd7cc2c"
format_stamp: "Formatted at 2026-06-20 19:10:52 on dist-test-slave-n3fx"
server_key: "591790b5684535df6290e88f470e684a"
server_key_iv: "0202de4dcd15ae657e522a0ddc3f4df1"
server_key_version: "encryptionkey@0"
I20260620 19:10:52.820060 19888 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 19:10:52.822715 19902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:52.824121 19888 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 19:10:52.824292 19888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-2/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-2/wal
uuid: "449ad3709aca4a27b7012df6bbd7cc2c"
format_stamp: "Formatted at 2026-06-20 19:10:52 on dist-test-slave-n3fx"
server_key: "591790b5684535df6290e88f470e684a"
server_key_iv: "0202de4dcd15ae657e522a0ddc3f4df1"
server_key_version: "encryptionkey@0"
I20260620 19:10:52.824421 19888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:52.829200 19888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:52.829944 19888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:52.830119 19888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:52.830803 19888 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:10:52.832015 19888 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:10:52.832070 19888 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:52.832110 19888 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:10:52.832129 19888 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:52.840704 19888 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.67:33931
I20260620 19:10:52.840721 20015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.67:33931 every 8 connection(s)
I20260620 19:10:52.841655 19888 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
I20260620 19:10:52.845615 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 19888
I20260620 19:10:52.845752 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982595231230-9837-0/minicluster-data/ts-2/wal/instance
I20260620 19:10:52.846041 9837 external_mini_cluster.cc:1428] Setting key 733dba9f426f1ff548bac2a56d244260
I20260620 19:10:52.848294 19605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:52.843540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.67:49407 (local address 127.9.155.126:41791)
0620 19:10:52.843684 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:52.843687 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:52.844839 (+ 1152us) server_negotiation.cc:407] Connection header received
0620 19:10:52.846365 (+ 1526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:52.846376 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:52.846480 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:52.846565 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:52.847812 (+ 1247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:52.847815 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:52.847817 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:10:52.847828 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:52.847849 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:10:52.847857 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:52.847931 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:52.848090 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 19:10:52.848134 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 19:10:52.849182 20018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:52.843986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:41791 (local address 127.9.155.67:49407)
0620 19:10:52.844688 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:52.844730 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:52.846117 (+ 1387us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:52.846754 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:52.846761 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:52.846863 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:52.847523 (+ 660us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:10:52.847525 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:52.847578 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:10:52.847668 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:52.847935 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:52.847942 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:52.847948 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 19:10:52.848118 (+ 170us) client_negotiation.cc:240] Negotiation successful
0620 19:10:52.848393 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":164,"threads_started":1}
I20260620 19:10:52.850198 20016 heartbeater.cc:347] Connected to a master server at 127.9.155.126:41791
I20260620 19:10:52.850428 20016 heartbeater.cc:464] Registering TS with master...
I20260620 19:10:52.850842 20016 heartbeater.cc:511] Master 127.9.155.126:41791 requested a full tablet report, sending...
I20260620 19:10:52.851888 19566 ts_manager.cc:194] Registered new tserver with Master: 449ad3709aca4a27b7012df6bbd7cc2c (127.9.155.67:33931)
I20260620 19:10:52.860301 9837 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 19:10:52.863013 19605 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:10:52.862243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34612 (local address 127.9.155.126:41791)
0620 19:10:52.862368 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:52.862371 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:52.862513 (+ 142us) server_negotiation.cc:407] Connection header received
0620 19:10:52.862619 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:52.862621 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:52.862656 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:52.862723 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:52.862891 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34612: BlockingRecv error: recv got EOF from 127.0.0.1:34612 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 19:10:52.863256 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 19626
I20260620 19:10:52.868462 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 19757
I20260620 19:10:52.873296 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 19888
I20260620 19:10:52.877916 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 19535
2026-06-20T19:10:52Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (831 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:10:52 dist-test-slave-n3fx krb5kdc[20026](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:10:52 dist-test-slave-n3fx krb5kdc[20026](info): set up 2 sockets
Jun 20 19:10:52 dist-test-slave-n3fx krb5kdc[20026](info): commencing operation
krb5kdc: starting...
W20260620 19:10:53.849016 9837 mini_kdc.cc:121] Time spent starting KDC: real 0.952s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/krb5kdc/test-user.keytab.
Jun 20 19:10:53 dist-test-slave-n3fx krb5kdc[20026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:10:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:10:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.155.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.155.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.155.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab.
Entry for principal kudu/127.9.155.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab.
I20260620 19:10:53.967357 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:44011
--webserver_interface=127.9.155.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab
--principal=kudu/127.9.155.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:37525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.155.126:44011
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:10:54.082139 20039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:54.082437 20039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:54.082499 20039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:54.086654 20039 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:10:54.086733 20039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:54.086759 20039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:54.086779 20039 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:10:54.086800 20039 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:10:54.092242 20039 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:37525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.155.126:44011
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab
--principal=kudu/127.9.155.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:44011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.155.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20039
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:10:54.093268 20039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:54.094256 20039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:10:54.102663 20044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:54.102717 20039 server_base.cc:1034] running on GCE node
W20260620 19:10:54.102674 20047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:54.102654 20045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:54.103518 20039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:54.104630 20039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:54.105793 20039 hybrid_clock.cc:648] HybridClock initialized: now 1781982654105777 us; error 32 us; skew 500 ppm
Jun 20 19:10:54 dist-test-slave-n3fx krb5kdc[20026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982654, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:54.108662 20039 init.cc:377] Logged in from keytab as kudu/127.9.155.126@KRBTEST.COM (short username kudu)
I20260620 19:10:54.109570 20039 webserver.cc:460] Webserver started at http://127.9.155.126:36323/ using document root <none> and password file <none>
I20260620 19:10:54.110112 20039 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:54.110175 20039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:54.110371 20039 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:10:54.112528 20039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/minicluster-data/master-0/data/instance:
uuid: "3ced55e2ed6b48a6853b70f5a8a4ea2b"
format_stamp: "Formatted at 2026-06-20 19:10:54 on dist-test-slave-n3fx"
server_key: "c68da5ce1698f5ccb28f70495b43b399"
server_key_iv: "894df499d88120c58a9f62d917e82ecf"
server_key_version: "encryptionkey@0"
I20260620 19:10:54.113101 20039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/minicluster-data/master-0/wal/instance:
uuid: "3ced55e2ed6b48a6853b70f5a8a4ea2b"
format_stamp: "Formatted at 2026-06-20 19:10:54 on dist-test-slave-n3fx"
server_key: "c68da5ce1698f5ccb28f70495b43b399"
server_key_iv: "894df499d88120c58a9f62d917e82ecf"
server_key_version: "encryptionkey@0"
I20260620 19:10:54.117118 20039 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 19:10:54.119812 20054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:54.121385 20039 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:10:54.121523 20039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/minicluster-data/master-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/minicluster-data/master-0/wal
uuid: "3ced55e2ed6b48a6853b70f5a8a4ea2b"
format_stamp: "Formatted at 2026-06-20 19:10:54 on dist-test-slave-n3fx"
server_key: "c68da5ce1698f5ccb28f70495b43b399"
server_key_iv: "894df499d88120c58a9f62d917e82ecf"
server_key_version: "encryptionkey@0"
I20260620 19:10:54.121662 20039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:54.160920 20039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:54.164202 20039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:54.164408 20039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:54.171310 20039 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.126:44011
I20260620 19:10:54.171643 20106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.126:44011 every 8 connection(s)
I20260620 19:10:54.172437 20039 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
I20260620 19:10:54.174100 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 20039
I20260620 19:10:54.174216 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/minicluster-data/master-0/wal/instance
I20260620 19:10:54.174508 9837 external_mini_cluster.cc:1428] Setting key eca78fe43cb2dfe698a55a63716999b3
I20260620 19:10:54.175637 20107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:54.181222 20107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ced55e2ed6b48a6853b70f5a8a4ea2b: Bootstrap starting.
Jun 20 19:10:54 dist-test-slave-n3fx krb5kdc[20026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:10:54.183630 20107 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ced55e2ed6b48a6853b70f5a8a4ea2b: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:54.184396 20107 log.cc:824] T 00000000000000000000000000000000 P 3ced55e2ed6b48a6853b70f5a8a4ea2b: Log is configured to *not* fsync() on all Append() calls
I20260620 19:10:54.186389 20107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ced55e2ed6b48a6853b70f5a8a4ea2b: No bootstrap required, opened a new log
I20260620 19:10:54.188138 20110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:54.175993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43578 (local address 127.9.155.126:44011)
0620 19:10:54.176486 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:54.176502 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:54.176539 (+ 37us) server_negotiation.cc:407] Connection header received
0620 19:10:54.177175 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:54.177191 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:54.177522 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:54.177820 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:54.178608 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:54.179601 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:54.180490 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:54.180941 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:54.181081 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:54.183615 (+ 2534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:54.183620 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:54.183634 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:54.183663 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:54.185484 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:54.185898 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:54.185904 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:54.185911 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:54.186030 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:54.186297 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:54.186300 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:54.186302 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:54.186722 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:54.186905 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:54.187159 (+ 254us) server_negotiation.cc:299] Negotiation successful
0620 19:10:54.187440 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":160,"threads_started":1}
I20260620 19:10:54.190253 20107 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ced55e2ed6b48a6853b70f5a8a4ea2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ced55e2ed6b48a6853b70f5a8a4ea2b" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 44011 } }
I20260620 19:10:54.190529 20107 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ced55e2ed6b48a6853b70f5a8a4ea2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:54.190589 20107 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ced55e2ed6b48a6853b70f5a8a4ea2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ced55e2ed6b48a6853b70f5a8a4ea2b, State: Initialized, Role: FOLLOWER
I20260620 19:10:54.190999 20107 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ced55e2ed6b48a6853b70f5a8a4ea2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ced55e2ed6b48a6853b70f5a8a4ea2b" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 44011 } }
I20260620 19:10:54.191128 20107 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3ced55e2ed6b48a6853b70f5a8a4ea2b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:10:54.191191 20107 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ced55e2ed6b48a6853b70f5a8a4ea2b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:10:54.191286 20107 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ced55e2ed6b48a6853b70f5a8a4ea2b [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:54.192209 20107 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ced55e2ed6b48a6853b70f5a8a4ea2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ced55e2ed6b48a6853b70f5a8a4ea2b" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 44011 } }
I20260620 19:10:54.192593 20107 leader_election.cc:302] T 00000000000000000000000000000000 P 3ced55e2ed6b48a6853b70f5a8a4ea2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ced55e2ed6b48a6853b70f5a8a4ea2b; no voters:
I20260620 19:10:54.192912 20107 leader_election.cc:288] T 00000000000000000000000000000000 P 3ced55e2ed6b48a6853b70f5a8a4ea2b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:10:54.193007 20112 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ced55e2ed6b48a6853b70f5a8a4ea2b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:10:54.193221 20112 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ced55e2ed6b48a6853b70f5a8a4ea2b [term 1 LEADER]: Becoming Leader. State: Replica: 3ced55e2ed6b48a6853b70f5a8a4ea2b, State: Running, Role: LEADER
I20260620 19:10:54.193627 20112 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ced55e2ed6b48a6853b70f5a8a4ea2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ced55e2ed6b48a6853b70f5a8a4ea2b" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 44011 } }
I20260620 19:10:54.194118 20107 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ced55e2ed6b48a6853b70f5a8a4ea2b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:10:54.195276 20114 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ced55e2ed6b48a6853b70f5a8a4ea2b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ced55e2ed6b48a6853b70f5a8a4ea2b. Latest consensus state: current_term: 1 leader_uuid: "3ced55e2ed6b48a6853b70f5a8a4ea2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ced55e2ed6b48a6853b70f5a8a4ea2b" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 44011 } } }
I20260620 19:10:54.195408 20114 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ced55e2ed6b48a6853b70f5a8a4ea2b [sys.catalog]: This master's current role is: LEADER
I20260620 19:10:54.195782 20121 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:10:54.195758 20113 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ced55e2ed6b48a6853b70f5a8a4ea2b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3ced55e2ed6b48a6853b70f5a8a4ea2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ced55e2ed6b48a6853b70f5a8a4ea2b" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 44011 } } }
I20260620 19:10:54.195914 20113 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ced55e2ed6b48a6853b70f5a8a4ea2b [sys.catalog]: This master's current role is: LEADER
I20260620 19:10:54.198546 20121 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:10:54.204525 20121 catalog_manager.cc:1347] Generated new cluster ID: 194b912e848b4731abd5037a8e8a4941
I20260620 19:10:54.204608 20121 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:10:54.213012 20121 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:10:54.214260 20121 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:10:54.227896 20121 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3ced55e2ed6b48a6853b70f5a8a4ea2b: Generated new TSK 0
I20260620 19:10:54.228775 20121 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.155.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.155.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.155.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.65.keytab.
Entry for principal kudu/127.9.155.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.65.keytab.
I20260620 19:10:54.254917 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.65:0
--local_ip_for_outbound_sockets=127.9.155.65
--webserver_interface=127.9.155.65
--webserver_port=0
--tserver_master_addrs=127.9.155.126:44011
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.65.keytab
--principal=kudu/127.9.155.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:37525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:10:54.369526 20133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:54.369817 20133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:54.369880 20133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:54.374147 20133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:54.374223 20133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:54.374310 20133 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.65
I20260620 19:10:54.379874 20133 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:37525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.65.keytab
--principal=kudu/127.9.155.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.155.65
--webserver_port=0
--tserver_master_addrs=127.9.155.126:44011
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20133
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.65
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:10:54.381098 20133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:54.382125 20133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:10:54.391877 20141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:54.391845 20138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:54.391935 20133 server_base.cc:1034] running on GCE node
W20260620 19:10:54.391845 20139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:54.392505 20133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:54.393209 20133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:54.394385 20133 hybrid_clock.cc:648] HybridClock initialized: now 1781982654394355 us; error 38 us; skew 500 ppm
Jun 20 19:10:54 dist-test-slave-n3fx krb5kdc[20026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982654, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:54.397090 20133 init.cc:377] Logged in from keytab as kudu/127.9.155.65@KRBTEST.COM (short username kudu)
I20260620 19:10:54.397941 20133 webserver.cc:460] Webserver started at http://127.9.155.65:41111/ using document root <none> and password file <none>
I20260620 19:10:54.398453 20133 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:54.398510 20133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:54.398671 20133 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:10:54.400673 20133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/minicluster-data/ts-0/data/instance:
uuid: "8ff79f8bd6ab47f9bfbc8fadd34fbb00"
format_stamp: "Formatted at 2026-06-20 19:10:54 on dist-test-slave-n3fx"
server_key: "ad54432ebdc9a7af4ca5912f71ac4886"
server_key_iv: "76ea82d41b0d5c7ca574e4f0a4167887"
server_key_version: "encryptionkey@0"
I20260620 19:10:54.401167 20133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/minicluster-data/ts-0/wal/instance:
uuid: "8ff79f8bd6ab47f9bfbc8fadd34fbb00"
format_stamp: "Formatted at 2026-06-20 19:10:54 on dist-test-slave-n3fx"
server_key: "ad54432ebdc9a7af4ca5912f71ac4886"
server_key_iv: "76ea82d41b0d5c7ca574e4f0a4167887"
server_key_version: "encryptionkey@0"
I20260620 19:10:54.404901 20133 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 19:10:54.407797 20148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:54.409365 20133 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:10:54.409536 20133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/minicluster-data/ts-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/minicluster-data/ts-0/wal
uuid: "8ff79f8bd6ab47f9bfbc8fadd34fbb00"
format_stamp: "Formatted at 2026-06-20 19:10:54 on dist-test-slave-n3fx"
server_key: "ad54432ebdc9a7af4ca5912f71ac4886"
server_key_iv: "76ea82d41b0d5c7ca574e4f0a4167887"
server_key_version: "encryptionkey@0"
I20260620 19:10:54.409668 20133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:54.421998 20133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:54.425536 20133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:54.425727 20133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:54.426345 20133 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:10:54.427194 20133 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:10:54.427242 20133 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:54.427294 20133 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:10:54.427325 20133 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:54.435696 20133 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.65:37433
I20260620 19:10:54.435745 20261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.65:37433 every 8 connection(s)
I20260620 19:10:54.436609 20133 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
I20260620 19:10:54.441473 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 20133
I20260620 19:10:54.441573 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/minicluster-data/ts-0/wal/instance
I20260620 19:10:54.441830 9837 external_mini_cluster.cc:1428] Setting key 877e690497e38d85668fbb055b8662ac
Jun 20 19:10:54 dist-test-slave-n3fx krb5kdc[20026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982654, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.65@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.155.66@KRBTEST.COM; defaulting to no policy
I20260620 19:10:54.450498 20110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:54.438435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.65:35547 (local address 127.9.155.126:44011)
0620 19:10:54.438568 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:54.438571 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:54.439372 (+ 801us) server_negotiation.cc:407] Connection header received
0620 19:10:54.440679 (+ 1307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:54.440683 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:54.440741 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:54.440814 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:54.442018 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:54.442621 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:54.444086 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:54.444400 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:54.444505 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:54.446957 (+ 2452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:54.446961 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:54.446964 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:54.446990 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:54.448444 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:54.448947 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:54.448951 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:54.448953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:54.449029 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:54.449341 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:54.449345 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:54.449347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:54.449533 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:54.449618 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:54.450126 (+ 508us) server_negotiation.cc:299] Negotiation successful
0620 19:10:54.450264 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 19:10:54.451249 20264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:54.438695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:44011 (local address 127.9.155.65:35547)
0620 19:10:54.439160 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:54.439202 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:54.440475 (+ 1273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:54.440990 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:54.440997 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:54.441366 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:54.441877 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:54.441897 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:54.442732 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:54.442736 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:54.443960 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:54.443971 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:54.444547 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:54.444551 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:54.444749 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:54.445379 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:54.445395 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:54.446785 (+ 1390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:54.448554 (+ 1769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:54.448560 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:54.448577 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:54.448855 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:54.449111 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:54.449114 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:54.449116 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:54.449252 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:54.449629 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:54.449636 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:54.449900 (+ 264us) client_negotiation.cc:769] Sending connection context
0620 19:10:54.450071 (+ 171us) client_negotiation.cc:240] Negotiation successful
0620 19:10:54.450420 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":119,"threads_started":1}
I20260620 19:10:54.452505 20262 heartbeater.cc:347] Connected to a master server at 127.9.155.126:44011
I20260620 19:10:54.452751 20262 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.155.66@KRBTEST.COM" created.
I20260620 19:10:54.453260 20262 heartbeater.cc:511] Master 127.9.155.126:44011 requested a full tablet report, sending...
I20260620 19:10:54.454941 20068 ts_manager.cc:194] Registered new tserver with Master: 8ff79f8bd6ab47f9bfbc8fadd34fbb00 (127.9.155.65:37433)
I20260620 19:10:54.456333 20068 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.155.65@KRBTEST.COM'} at 127.9.155.65:35547
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.155.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.66.keytab.
Entry for principal kudu/127.9.155.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.66.keytab.
I20260620 19:10:54.468765 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.66:0
--local_ip_for_outbound_sockets=127.9.155.66
--webserver_interface=127.9.155.66
--webserver_port=0
--tserver_master_addrs=127.9.155.126:44011
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.66.keytab
--principal=kudu/127.9.155.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:37525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:10:54.576725 20267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:54.577165 20267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:54.577237 20267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:54.581430 20267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:54.581506 20267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:54.581591 20267 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.66
I20260620 19:10:54.587039 20267 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:37525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.66.keytab
--principal=kudu/127.9.155.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.155.66
--webserver_port=0
--tserver_master_addrs=127.9.155.126:44011
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20267
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.66
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:10:54.588173 20267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:54.589284 20267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:10:54.598912 20275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:54.599006 20273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:54.598932 20272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:54.598995 20267 server_base.cc:1034] running on GCE node
I20260620 19:10:54.599552 20267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:54.600237 20267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:54.601441 20267 hybrid_clock.cc:648] HybridClock initialized: now 1781982654601423 us; error 32 us; skew 500 ppm
Jun 20 19:10:54 dist-test-slave-n3fx krb5kdc[20026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982654, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:54.604048 20267 init.cc:377] Logged in from keytab as kudu/127.9.155.66@KRBTEST.COM (short username kudu)
I20260620 19:10:54.604918 20267 webserver.cc:460] Webserver started at http://127.9.155.66:43145/ using document root <none> and password file <none>
I20260620 19:10:54.605429 20267 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:54.605489 20267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:54.605648 20267 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:10:54.607606 20267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/minicluster-data/ts-1/data/instance:
uuid: "98a1172a78e34632ad35688f1192a6af"
format_stamp: "Formatted at 2026-06-20 19:10:54 on dist-test-slave-n3fx"
server_key: "9119d3d84dcb4dabdfad4ee1049c72ec"
server_key_iv: "ff7f5193b461db26dda971413214d35d"
server_key_version: "encryptionkey@0"
I20260620 19:10:54.608095 20267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/minicluster-data/ts-1/wal/instance:
uuid: "98a1172a78e34632ad35688f1192a6af"
format_stamp: "Formatted at 2026-06-20 19:10:54 on dist-test-slave-n3fx"
server_key: "9119d3d84dcb4dabdfad4ee1049c72ec"
server_key_iv: "ff7f5193b461db26dda971413214d35d"
server_key_version: "encryptionkey@0"
I20260620 19:10:54.611979 20267 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260620 19:10:54.614799 20282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:54.616173 20267 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 19:10:54.616343 20267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/minicluster-data/ts-1/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/minicluster-data/ts-1/wal
uuid: "98a1172a78e34632ad35688f1192a6af"
format_stamp: "Formatted at 2026-06-20 19:10:54 on dist-test-slave-n3fx"
server_key: "9119d3d84dcb4dabdfad4ee1049c72ec"
server_key_iv: "ff7f5193b461db26dda971413214d35d"
server_key_version: "encryptionkey@0"
I20260620 19:10:54.616473 20267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:54.663430 20267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:54.666481 20267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:54.666648 20267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:54.667236 20267 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:10:54.668025 20267 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:10:54.668071 20267 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:54.668109 20267 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:10:54.668123 20267 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:54.676010 20267 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.66:40949
I20260620 19:10:54.676043 20395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.66:40949 every 8 connection(s)
I20260620 19:10:54.676841 20267 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
I20260620 19:10:54.685202 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 20267
I20260620 19:10:54.685374 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/minicluster-data/ts-1/wal/instance
I20260620 19:10:54.685659 9837 external_mini_cluster.cc:1428] Setting key bb33f9f267e16781f58764cb2eb658c6
Jun 20 19:10:54 dist-test-slave-n3fx krb5kdc[20026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982654, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.66@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:10:54.690589 20110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:54.678512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.66:45847 (local address 127.9.155.126:44011)
0620 19:10:54.678668 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:54.678673 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:54.679494 (+ 821us) server_negotiation.cc:407] Connection header received
0620 19:10:54.680785 (+ 1291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:54.680788 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:54.680844 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:54.680944 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:54.682094 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:54.682628 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:54.683808 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:54.684051 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:54.684135 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:54.686818 (+ 2683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:54.686822 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:54.686824 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:54.686849 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:54.688397 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:54.689012 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:54.689015 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:54.689016 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:54.689096 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:54.689452 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:54.689454 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:54.689456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:54.689636 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:54.689710 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:54.690342 (+ 632us) server_negotiation.cc:299] Negotiation successful
0620 19:10:54.690468 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 19:10:54.691704 20398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:54.678911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:44011 (local address 127.9.155.66:45847)
0620 19:10:54.679336 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:54.679383 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:54.680587 (+ 1204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:54.681110 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:54.681117 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:54.681494 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:54.681972 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:54.681985 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:54.682747 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:54.682750 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:54.683685 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:54.683695 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:54.684170 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:54.684173 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:54.684410 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:54.684939 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:54.684955 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:54.686656 (+ 1701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:54.688513 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:54.688518 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:54.688532 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:54.688902 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:54.689177 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:54.689180 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:54.689182 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:54.689370 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:54.689726 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:54.689733 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:54.690070 (+ 337us) client_negotiation.cc:769] Sending connection context
0620 19:10:54.690274 (+ 204us) client_negotiation.cc:240] Negotiation successful
0620 19:10:54.690619 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":110,"threads_started":1}
I20260620 19:10:54.692992 20396 heartbeater.cc:347] Connected to a master server at 127.9.155.126:44011
I20260620 19:10:54.693262 20396 heartbeater.cc:464] Registering TS with master...
I20260620 19:10:54.693785 20396 heartbeater.cc:511] Master 127.9.155.126:44011 requested a full tablet report, sending...
I20260620 19:10:54.694834 20068 ts_manager.cc:194] Registered new tserver with Master: 98a1172a78e34632ad35688f1192a6af (127.9.155.66:40949)
WARNING: no policy specified for kudu/127.9.155.67@KRBTEST.COM; defaulting to no policy
I20260620 19:10:54.695708 20068 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.155.66@KRBTEST.COM'} at 127.9.155.66:45847
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.155.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.155.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.67.keytab.
Entry for principal kudu/127.9.155.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.67.keytab.
I20260620 19:10:54.713503 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.67:0
--local_ip_for_outbound_sockets=127.9.155.67
--webserver_interface=127.9.155.67
--webserver_port=0
--tserver_master_addrs=127.9.155.126:44011
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.67.keytab
--principal=kudu/127.9.155.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:37525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:10:54.824645 20401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:10:54.824931 20401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:10:54.824976 20401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:10:54.829195 20401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:10:54.829262 20401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:10:54.829335 20401 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.67
I20260620 19:10:54.834722 20401 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:37525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.67.keytab
--principal=kudu/127.9.155.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.155.67
--webserver_port=0
--tserver_master_addrs=127.9.155.126:44011
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20401
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.67
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:10:54.835813 20401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:10:54.836856 20401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:10:54.846689 20406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:54.846787 20401 server_base.cc:1034] running on GCE node
W20260620 19:10:54.846804 20409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:10:54.846689 20407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:10:54.847461 20401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:10:54.848124 20401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:10:54.849272 20401 hybrid_clock.cc:648] HybridClock initialized: now 1781982654849231 us; error 53 us; skew 500 ppm
Jun 20 19:10:54 dist-test-slave-n3fx krb5kdc[20026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982654, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:10:54.852095 20401 init.cc:377] Logged in from keytab as kudu/127.9.155.67@KRBTEST.COM (short username kudu)
I20260620 19:10:54.853068 20401 webserver.cc:460] Webserver started at http://127.9.155.67:43879/ using document root <none> and password file <none>
I20260620 19:10:54.853631 20401 fs_manager.cc:359] Metadata directory not provided
I20260620 19:10:54.853691 20401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:10:54.853861 20401 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:10:54.855926 20401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/minicluster-data/ts-2/data/instance:
uuid: "126d56d590ae455cb0651a29df830c7a"
format_stamp: "Formatted at 2026-06-20 19:10:54 on dist-test-slave-n3fx"
server_key: "b1f0b619475d252f2896e79980b3c779"
server_key_iv: "52b73a9910740a95ac4da444a6b92c28"
server_key_version: "encryptionkey@0"
I20260620 19:10:54.856516 20401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/minicluster-data/ts-2/wal/instance:
uuid: "126d56d590ae455cb0651a29df830c7a"
format_stamp: "Formatted at 2026-06-20 19:10:54 on dist-test-slave-n3fx"
server_key: "b1f0b619475d252f2896e79980b3c779"
server_key_iv: "52b73a9910740a95ac4da444a6b92c28"
server_key_version: "encryptionkey@0"
I20260620 19:10:54.860541 20401 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 19:10:54.863168 20416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:54.864665 20401 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:10:54.864774 20401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/minicluster-data/ts-2/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/minicluster-data/ts-2/wal
uuid: "126d56d590ae455cb0651a29df830c7a"
format_stamp: "Formatted at 2026-06-20 19:10:54 on dist-test-slave-n3fx"
server_key: "b1f0b619475d252f2896e79980b3c779"
server_key_iv: "52b73a9910740a95ac4da444a6b92c28"
server_key_version: "encryptionkey@0"
I20260620 19:10:54.864881 20401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:10:54.890491 20401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:10:54.893829 20401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:10:54.894048 20401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:10:54.894659 20401 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:10:54.895571 20401 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:10:54.895622 20401 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:54.895661 20401 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:10:54.895676 20401 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:10:54.904525 20401 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.67:45815
I20260620 19:10:54.904563 20529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.67:45815 every 8 connection(s)
I20260620 19:10:54.905501 20401 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
I20260620 19:10:54.910427 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 20401
I20260620 19:10:54.910538 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/minicluster-data/ts-2/wal/instance
I20260620 19:10:54.910789 9837 external_mini_cluster.cc:1428] Setting key 9bda9c336d770f0502bccdb3aa99ed53
Jun 20 19:10:54 dist-test-slave-n3fx krb5kdc[20026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982654, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.67@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:10:54.919816 20110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:54.907377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.67:45595 (local address 127.9.155.126:44011)
0620 19:10:54.907538 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:54.907542 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:54.908405 (+ 863us) server_negotiation.cc:407] Connection header received
0620 19:10:54.909839 (+ 1434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:54.909843 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:54.909905 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:54.910009 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:54.911407 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:54.912099 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:54.913316 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:54.913541 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:54.913628 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:54.916166 (+ 2538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:54.916169 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:54.916171 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:54.916197 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:54.917669 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:54.918220 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:54.918224 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:54.918226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:54.918308 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:54.918637 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:54.918640 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:54.918642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:54.918857 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:54.918945 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:54.919483 (+ 538us) server_negotiation.cc:299] Negotiation successful
0620 19:10:54.919617 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 19:10:54.920748 20532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:54.907747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:44011 (local address 127.9.155.67:45595)
0620 19:10:54.908255 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:54.908304 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:54.909599 (+ 1295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:54.910211 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:54.910219 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:54.910669 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:10:54.911258 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:54.911282 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:54.912243 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:54.912247 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:54.913204 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:54.913214 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:54.913683 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:54.913687 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:54.913856 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:54.914469 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:10:54.914488 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:10:54.915976 (+ 1488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:10:54.917795 (+ 1819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:54.917802 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:54.917821 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:54.918116 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:54.918394 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:10:54.918397 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:10:54.918399 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:10:54.918548 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:10:54.918957 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:10:54.918965 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:10:54.919232 (+ 267us) client_negotiation.cc:769] Sending connection context
0620 19:10:54.919425 (+ 193us) client_negotiation.cc:240] Negotiation successful
0620 19:10:54.919778 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":124,"threads_started":1}
I20260620 19:10:54.921880 20530 heartbeater.cc:347] Connected to a master server at 127.9.155.126:44011
I20260620 19:10:54.922103 20530 heartbeater.cc:464] Registering TS with master...
I20260620 19:10:54.922569 20530 heartbeater.cc:511] Master 127.9.155.126:44011 requested a full tablet report, sending...
I20260620 19:10:54.923671 20068 ts_manager.cc:194] Registered new tserver with Master: 126d56d590ae455cb0651a29df830c7a (127.9.155.67:45815)
I20260620 19:10:54.924412 20068 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.155.67@KRBTEST.COM'} at 127.9.155.67:45595
I20260620 19:10:54.924753 9837 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:10:54.933979 20110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:54.926776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43580 (local address 127.9.155.126:44011)
0620 19:10:54.926893 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:54.926898 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:54.927146 (+ 248us) server_negotiation.cc:407] Connection header received
0620 19:10:54.927287 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:54.927290 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:54.927333 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:54.927411 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:10:54.928059 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:54.928627 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:54.929544 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:54.929759 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:54.929839 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:54.931213 (+ 1374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:10:54.931216 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:10:54.931218 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:10:54.931241 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:10:54.932503 (+ 1262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:54.932895 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:54.932898 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:54.932899 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:54.932971 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:10:54.933184 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:10:54.933186 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:10:54.933187 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:10:54.933407 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:10:54.933508 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:54.933706 (+ 198us) server_negotiation.cc:299] Negotiation successful
0620 19:10:54.933803 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 19:10:54.936908 20068 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43580:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:10:54.940017 20068 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:10:54.954886 20546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:54.949740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56084 (local address 127.9.155.66:40949)
0620 19:10:54.950380 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:54.950392 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:54.950411 (+ 19us) server_negotiation.cc:407] Connection header received
0620 19:10:54.950479 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:54.950484 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:54.950656 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:54.950774 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:10:54.951007 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:54.952045 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:54.953375 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:54.954224 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:54.954337 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:54.954493 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:54.954665 (+ 172us) server_negotiation.cc:299] Negotiation successful
0620 19:10:54.954748 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":115,"threads_started":1}
I20260620 19:10:54.955173 20542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:54.948825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42156 (local address 127.9.155.67:45815)
0620 19:10:54.951068 (+ 2243us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:54.951077 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:54.951094 (+ 17us) server_negotiation.cc:407] Connection header received
0620 19:10:54.951151 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:54.951154 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:54.951277 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:54.951400 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:10:54.951581 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:54.952582 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:54.953824 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:54.954405 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:54.954501 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:54.954635 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:54.954920 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 19:10:54.955054 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2147,"thread_start_us":124,"threads_started":1}
I20260620 19:10:54.954927 20545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:54.949359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.66:40949 (local address 127.0.0.1:56084)
0620 19:10:54.950095 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:54.950107 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:54.950178 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:54.950762 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:54.950765 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:54.950781 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:10:54.950874 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:54.950880 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:54.952208 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:54.952242 (+ 34us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:54.953259 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:54.953268 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:54.954358 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:54.954362 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:54.954474 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:54.954520 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 19:10:54.954633 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 19:10:54.954767 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":682,"thread_start_us":90,"threads_started":1}
I20260620 19:10:54.955505 20543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:54.948648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.67:45815 (local address 127.0.0.1:42156)
0620 19:10:54.949577 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:54.949631 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:54.949764 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:54.951392 (+ 1628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:54.951395 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:54.951411 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:10:54.951497 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:54.951503 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:54.952723 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:54.952726 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:54.953720 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:54.953727 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:54.954690 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:54.954693 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:54.954777 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:54.954796 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 19:10:54.954866 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 19:10:54.954925 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":753,"mutex_wait_us":25,"spinlock_wait_cycles":6016,"thread_start_us":99,"threads_started":1}
I20260620 19:10:54.955613 20544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:54.949133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52758 (local address 127.9.155.65:37433)
0620 19:10:54.949427 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:54.949443 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:54.949895 (+ 452us) server_negotiation.cc:407] Connection header received
0620 19:10:54.949970 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:54.949974 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:54.950176 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:54.950298 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:10:54.950607 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:54.951823 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:54.953486 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:54.954230 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:54.954337 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:54.954493 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:54.955354 (+ 861us) server_negotiation.cc:299] Negotiation successful
0620 19:10:54.955469 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":79,"threads_started":1}
I20260620 19:10:54.955703 20541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:54.948964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.65:37433 (local address 127.0.0.1:52758)
0620 19:10:54.949595 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:54.949631 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:54.949764 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:54.950330 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:54.950335 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:54.950357 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:10:54.950504 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:54.950510 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:54.951933 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:54.951936 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:54.953358 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:54.953368 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:54.955090 (+ 1722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:54.955094 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:54.955176 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:54.955197 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 19:10:54.955260 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 19:10:54.955328 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"mutex_wait_us":1,"spinlock_wait_cycles":2688,"thread_start_us":87,"threads_started":1}
I20260620 19:10:54.958331 20464 tablet_service.cc:1467] Processing CreateTablet for tablet fca4a269b5af4fcd92bcedc7630a424d (DEFAULT_TABLE table=test-table [id=3c494a01444642c18ae3282ad2ea558e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:10:54.958640 20330 tablet_service.cc:1467] Processing CreateTablet for tablet fca4a269b5af4fcd92bcedc7630a424d (DEFAULT_TABLE table=test-table [id=3c494a01444642c18ae3282ad2ea558e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:10:54.958638 20196 tablet_service.cc:1467] Processing CreateTablet for tablet fca4a269b5af4fcd92bcedc7630a424d (DEFAULT_TABLE table=test-table [id=3c494a01444642c18ae3282ad2ea558e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:10:54.959281 20464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fca4a269b5af4fcd92bcedc7630a424d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:54.959649 20330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fca4a269b5af4fcd92bcedc7630a424d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:54.959649 20196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fca4a269b5af4fcd92bcedc7630a424d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:10:54.966747 20548 tablet_bootstrap.cc:492] T fca4a269b5af4fcd92bcedc7630a424d P 98a1172a78e34632ad35688f1192a6af: Bootstrap starting.
I20260620 19:10:54.967316 20547 tablet_bootstrap.cc:492] T fca4a269b5af4fcd92bcedc7630a424d P 126d56d590ae455cb0651a29df830c7a: Bootstrap starting.
I20260620 19:10:54.967756 20549 tablet_bootstrap.cc:492] T fca4a269b5af4fcd92bcedc7630a424d P 8ff79f8bd6ab47f9bfbc8fadd34fbb00: Bootstrap starting.
I20260620 19:10:54.969573 20549 tablet_bootstrap.cc:654] T fca4a269b5af4fcd92bcedc7630a424d P 8ff79f8bd6ab47f9bfbc8fadd34fbb00: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:54.969658 20548 tablet_bootstrap.cc:654] T fca4a269b5af4fcd92bcedc7630a424d P 98a1172a78e34632ad35688f1192a6af: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:54.969807 20547 tablet_bootstrap.cc:654] T fca4a269b5af4fcd92bcedc7630a424d P 126d56d590ae455cb0651a29df830c7a: Neither blocks nor log segments found. Creating new log.
I20260620 19:10:54.970348 20549 log.cc:824] T fca4a269b5af4fcd92bcedc7630a424d P 8ff79f8bd6ab47f9bfbc8fadd34fbb00: Log is configured to *not* fsync() on all Append() calls
I20260620 19:10:54.970558 20547 log.cc:824] T fca4a269b5af4fcd92bcedc7630a424d P 126d56d590ae455cb0651a29df830c7a: Log is configured to *not* fsync() on all Append() calls
I20260620 19:10:54.970558 20548 log.cc:824] T fca4a269b5af4fcd92bcedc7630a424d P 98a1172a78e34632ad35688f1192a6af: Log is configured to *not* fsync() on all Append() calls
I20260620 19:10:54.972446 20549 tablet_bootstrap.cc:492] T fca4a269b5af4fcd92bcedc7630a424d P 8ff79f8bd6ab47f9bfbc8fadd34fbb00: No bootstrap required, opened a new log
I20260620 19:10:54.972524 20547 tablet_bootstrap.cc:492] T fca4a269b5af4fcd92bcedc7630a424d P 126d56d590ae455cb0651a29df830c7a: No bootstrap required, opened a new log
I20260620 19:10:54.972524 20548 tablet_bootstrap.cc:492] T fca4a269b5af4fcd92bcedc7630a424d P 98a1172a78e34632ad35688f1192a6af: No bootstrap required, opened a new log
I20260620 19:10:54.972657 20549 ts_tablet_manager.cc:1397] T fca4a269b5af4fcd92bcedc7630a424d P 8ff79f8bd6ab47f9bfbc8fadd34fbb00: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 19:10:54.972699 20548 ts_tablet_manager.cc:1397] T fca4a269b5af4fcd92bcedc7630a424d P 98a1172a78e34632ad35688f1192a6af: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 19:10:54.972699 20547 ts_tablet_manager.cc:1397] T fca4a269b5af4fcd92bcedc7630a424d P 126d56d590ae455cb0651a29df830c7a: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 19:10:54.976174 20549 raft_consensus.cc:348] T fca4a269b5af4fcd92bcedc7630a424d P 8ff79f8bd6ab47f9bfbc8fadd34fbb00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "126d56d590ae455cb0651a29df830c7a" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 45815 } } peers { permanent_uuid: "98a1172a78e34632ad35688f1192a6af" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40949 } } peers { permanent_uuid: "8ff79f8bd6ab47f9bfbc8fadd34fbb00" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 37433 } }
I20260620 19:10:54.976469 20549 raft_consensus.cc:374] T fca4a269b5af4fcd92bcedc7630a424d P 8ff79f8bd6ab47f9bfbc8fadd34fbb00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:54.976524 20549 raft_consensus.cc:729] T fca4a269b5af4fcd92bcedc7630a424d P 8ff79f8bd6ab47f9bfbc8fadd34fbb00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ff79f8bd6ab47f9bfbc8fadd34fbb00, State: Initialized, Role: FOLLOWER
I20260620 19:10:54.976558 20548 raft_consensus.cc:348] T fca4a269b5af4fcd92bcedc7630a424d P 98a1172a78e34632ad35688f1192a6af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "126d56d590ae455cb0651a29df830c7a" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 45815 } } peers { permanent_uuid: "98a1172a78e34632ad35688f1192a6af" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40949 } } peers { permanent_uuid: "8ff79f8bd6ab47f9bfbc8fadd34fbb00" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 37433 } }
I20260620 19:10:54.976558 20547 raft_consensus.cc:348] T fca4a269b5af4fcd92bcedc7630a424d P 126d56d590ae455cb0651a29df830c7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "126d56d590ae455cb0651a29df830c7a" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 45815 } } peers { permanent_uuid: "98a1172a78e34632ad35688f1192a6af" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40949 } } peers { permanent_uuid: "8ff79f8bd6ab47f9bfbc8fadd34fbb00" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 37433 } }
I20260620 19:10:54.976814 20548 raft_consensus.cc:374] T fca4a269b5af4fcd92bcedc7630a424d P 98a1172a78e34632ad35688f1192a6af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:54.976814 20547 raft_consensus.cc:374] T fca4a269b5af4fcd92bcedc7630a424d P 126d56d590ae455cb0651a29df830c7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:10:54.976867 20547 raft_consensus.cc:729] T fca4a269b5af4fcd92bcedc7630a424d P 126d56d590ae455cb0651a29df830c7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 126d56d590ae455cb0651a29df830c7a, State: Initialized, Role: FOLLOWER
I20260620 19:10:54.976877 20548 raft_consensus.cc:729] T fca4a269b5af4fcd92bcedc7630a424d P 98a1172a78e34632ad35688f1192a6af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98a1172a78e34632ad35688f1192a6af, State: Initialized, Role: FOLLOWER
I20260620 19:10:54.976995 20549 consensus_queue.cc:260] T fca4a269b5af4fcd92bcedc7630a424d P 8ff79f8bd6ab47f9bfbc8fadd34fbb00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "126d56d590ae455cb0651a29df830c7a" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 45815 } } peers { permanent_uuid: "98a1172a78e34632ad35688f1192a6af" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40949 } } peers { permanent_uuid: "8ff79f8bd6ab47f9bfbc8fadd34fbb00" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 37433 } }
I20260620 19:10:54.977293 20548 consensus_queue.cc:260] T fca4a269b5af4fcd92bcedc7630a424d P 98a1172a78e34632ad35688f1192a6af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "126d56d590ae455cb0651a29df830c7a" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 45815 } } peers { permanent_uuid: "98a1172a78e34632ad35688f1192a6af" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40949 } } peers { permanent_uuid: "8ff79f8bd6ab47f9bfbc8fadd34fbb00" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 37433 } }
I20260620 19:10:54.977293 20547 consensus_queue.cc:260] T fca4a269b5af4fcd92bcedc7630a424d P 126d56d590ae455cb0651a29df830c7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "126d56d590ae455cb0651a29df830c7a" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 45815 } } peers { permanent_uuid: "98a1172a78e34632ad35688f1192a6af" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40949 } } peers { permanent_uuid: "8ff79f8bd6ab47f9bfbc8fadd34fbb00" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 37433 } }
I20260620 19:10:54.977744 20262 heartbeater.cc:503] Master 127.9.155.126:44011 was elected leader, sending a full tablet report...
I20260620 19:10:54.978068 20549 ts_tablet_manager.cc:1428] T fca4a269b5af4fcd92bcedc7630a424d P 8ff79f8bd6ab47f9bfbc8fadd34fbb00: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 19:10:54.978230 20548 ts_tablet_manager.cc:1428] T fca4a269b5af4fcd92bcedc7630a424d P 98a1172a78e34632ad35688f1192a6af: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 19:10:54.978230 20547 ts_tablet_manager.cc:1428] T fca4a269b5af4fcd92bcedc7630a424d P 126d56d590ae455cb0651a29df830c7a: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 19:10:54.978566 20530 heartbeater.cc:503] Master 127.9.155.126:44011 was elected leader, sending a full tablet report...
I20260620 19:10:54.978575 20396 heartbeater.cc:503] Master 127.9.155.126:44011 was elected leader, sending a full tablet report...
I20260620 19:10:55.025843 20555 raft_consensus.cc:482] T fca4a269b5af4fcd92bcedc7630a424d P 98a1172a78e34632ad35688f1192a6af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:10:55.026090 20555 raft_consensus.cc:504] T fca4a269b5af4fcd92bcedc7630a424d P 98a1172a78e34632ad35688f1192a6af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "126d56d590ae455cb0651a29df830c7a" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 45815 } } peers { permanent_uuid: "98a1172a78e34632ad35688f1192a6af" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40949 } } peers { permanent_uuid: "8ff79f8bd6ab47f9bfbc8fadd34fbb00" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 37433 } }
I20260620 19:10:55.027496 20555 leader_election.cc:288] T fca4a269b5af4fcd92bcedc7630a424d P 98a1172a78e34632ad35688f1192a6af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 126d56d590ae455cb0651a29df830c7a (127.9.155.67:45815), 8ff79f8bd6ab47f9bfbc8fadd34fbb00 (127.9.155.65:37433)
I20260620 19:10:55.032802 20542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:55.027931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.66:58799 (local address 127.9.155.67:45815)
0620 19:10:55.028042 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:55.028046 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:55.028060 (+ 14us) server_negotiation.cc:407] Connection header received
0620 19:10:55.028183 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:55.028186 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:55.028256 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:55.028323 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:10:55.028712 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:55.029500 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:55.031300 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:55.032015 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:55.032173 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:55.032260 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:55.032555 (+ 295us) server_negotiation.cc:299] Negotiation successful
0620 19:10:55.032655 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 19:10:55.032794 20544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:55.028268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.66:50829 (local address 127.9.155.65:37433)
0620 19:10:55.028391 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:55.028396 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:55.028409 (+ 13us) server_negotiation.cc:407] Connection header received
0620 19:10:55.028489 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:55.028492 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:55.028530 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:55.028595 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:10:55.028849 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:55.029600 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:55.031300 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:55.032064 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:55.032185 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:55.032283 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:55.032580 (+ 297us) server_negotiation.cc:299] Negotiation successful
0620 19:10:55.032669 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 19:10:55.032796 20398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:55.027714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.67:45815 (local address 127.9.155.66:58799)
0620 19:10:55.027947 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:55.027962 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:55.028057 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:55.028440 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:55.028442 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:55.028458 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:10:55.028573 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:55.028579 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:55.029636 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:55.029639 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:55.031129 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:55.031142 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:55.032191 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:55.032196 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:55.032388 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:55.032405 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 19:10:55.032494 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 19:10:55.032612 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":145}
I20260620 19:10:55.032804 20556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:55.027714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.65:37433 (local address 127.9.155.66:50829)
0620 19:10:55.028304 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:55.028317 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:10:55.028395 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:10:55.028619 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:10:55.028622 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:10:55.028640 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:10:55.028751 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:55.028756 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:55.029703 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:55.029706 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:55.031126 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:10:55.031142 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:10:55.032191 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:10:55.032196 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:10:55.032356 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:55.032398 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 19:10:55.032526 (+ 128us) client_negotiation.cc:240] Negotiation successful
0620 19:10:55.032605 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"mutex_wait_us":240,"thread_start_us":130,"threads_started":1}
I20260620 19:10:55.033639 20216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fca4a269b5af4fcd92bcedc7630a424d" candidate_uuid: "98a1172a78e34632ad35688f1192a6af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ff79f8bd6ab47f9bfbc8fadd34fbb00" is_pre_election: true
I20260620 19:10:55.033638 20484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fca4a269b5af4fcd92bcedc7630a424d" candidate_uuid: "98a1172a78e34632ad35688f1192a6af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "126d56d590ae455cb0651a29df830c7a" is_pre_election: true
I20260620 19:10:55.034008 20484 raft_consensus.cc:2454] T fca4a269b5af4fcd92bcedc7630a424d P 126d56d590ae455cb0651a29df830c7a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98a1172a78e34632ad35688f1192a6af in term 0.
I20260620 19:10:55.034008 20216 raft_consensus.cc:2454] T fca4a269b5af4fcd92bcedc7630a424d P 8ff79f8bd6ab47f9bfbc8fadd34fbb00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98a1172a78e34632ad35688f1192a6af in term 0.
I20260620 19:10:55.034583 20286 leader_election.cc:302] T fca4a269b5af4fcd92bcedc7630a424d P 98a1172a78e34632ad35688f1192a6af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ff79f8bd6ab47f9bfbc8fadd34fbb00, 98a1172a78e34632ad35688f1192a6af; no voters:
I20260620 19:10:55.034925 20555 raft_consensus.cc:2781] T fca4a269b5af4fcd92bcedc7630a424d P 98a1172a78e34632ad35688f1192a6af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:10:55.035023 20555 raft_consensus.cc:482] T fca4a269b5af4fcd92bcedc7630a424d P 98a1172a78e34632ad35688f1192a6af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:10:55.035066 20555 raft_consensus.cc:3037] T fca4a269b5af4fcd92bcedc7630a424d P 98a1172a78e34632ad35688f1192a6af [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:55.036159 20555 raft_consensus.cc:504] T fca4a269b5af4fcd92bcedc7630a424d P 98a1172a78e34632ad35688f1192a6af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "126d56d590ae455cb0651a29df830c7a" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 45815 } } peers { permanent_uuid: "98a1172a78e34632ad35688f1192a6af" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40949 } } peers { permanent_uuid: "8ff79f8bd6ab47f9bfbc8fadd34fbb00" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 37433 } }
I20260620 19:10:55.036574 20555 leader_election.cc:288] T fca4a269b5af4fcd92bcedc7630a424d P 98a1172a78e34632ad35688f1192a6af [CANDIDATE]: Term 1 election: Requested vote from peers 126d56d590ae455cb0651a29df830c7a (127.9.155.67:45815), 8ff79f8bd6ab47f9bfbc8fadd34fbb00 (127.9.155.65:37433)
I20260620 19:10:55.037133 20484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fca4a269b5af4fcd92bcedc7630a424d" candidate_uuid: "98a1172a78e34632ad35688f1192a6af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "126d56d590ae455cb0651a29df830c7a"
I20260620 19:10:55.037133 20216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fca4a269b5af4fcd92bcedc7630a424d" candidate_uuid: "98a1172a78e34632ad35688f1192a6af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ff79f8bd6ab47f9bfbc8fadd34fbb00"
I20260620 19:10:55.037279 20484 raft_consensus.cc:3037] T fca4a269b5af4fcd92bcedc7630a424d P 126d56d590ae455cb0651a29df830c7a [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:55.037279 20216 raft_consensus.cc:3037] T fca4a269b5af4fcd92bcedc7630a424d P 8ff79f8bd6ab47f9bfbc8fadd34fbb00 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:10:55.038587 20216 raft_consensus.cc:2454] T fca4a269b5af4fcd92bcedc7630a424d P 8ff79f8bd6ab47f9bfbc8fadd34fbb00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98a1172a78e34632ad35688f1192a6af in term 1.
I20260620 19:10:55.038587 20484 raft_consensus.cc:2454] T fca4a269b5af4fcd92bcedc7630a424d P 126d56d590ae455cb0651a29df830c7a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98a1172a78e34632ad35688f1192a6af in term 1.
I20260620 19:10:55.038964 20283 leader_election.cc:302] T fca4a269b5af4fcd92bcedc7630a424d P 98a1172a78e34632ad35688f1192a6af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 126d56d590ae455cb0651a29df830c7a, 8ff79f8bd6ab47f9bfbc8fadd34fbb00, 98a1172a78e34632ad35688f1192a6af; no voters:
I20260620 19:10:55.039143 20555 raft_consensus.cc:2781] T fca4a269b5af4fcd92bcedc7630a424d P 98a1172a78e34632ad35688f1192a6af [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:10:55.039397 20555 raft_consensus.cc:686] T fca4a269b5af4fcd92bcedc7630a424d P 98a1172a78e34632ad35688f1192a6af [term 1 LEADER]: Becoming Leader. State: Replica: 98a1172a78e34632ad35688f1192a6af, State: Running, Role: LEADER
I20260620 19:10:55.039932 20555 consensus_queue.cc:237] T fca4a269b5af4fcd92bcedc7630a424d P 98a1172a78e34632ad35688f1192a6af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "126d56d590ae455cb0651a29df830c7a" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 45815 } } peers { permanent_uuid: "98a1172a78e34632ad35688f1192a6af" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40949 } } peers { permanent_uuid: "8ff79f8bd6ab47f9bfbc8fadd34fbb00" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 37433 } }
I20260620 19:10:55.043316 20071 catalog_manager.cc:5495] T fca4a269b5af4fcd92bcedc7630a424d P 98a1172a78e34632ad35688f1192a6af reported cstate change: term changed from 0 to 1, leader changed from <none> to 98a1172a78e34632ad35688f1192a6af (127.9.155.66). New cstate: current_term: 1 leader_uuid: "98a1172a78e34632ad35688f1192a6af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "126d56d590ae455cb0651a29df830c7a" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 45815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98a1172a78e34632ad35688f1192a6af" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40949 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ff79f8bd6ab47f9bfbc8fadd34fbb00" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 37433 } health_report { overall_health: UNKNOWN } } }
I20260620 19:10:55.056663 20546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:55.052760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56098 (local address 127.9.155.66:40949)
0620 19:10:55.052918 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 19:10:55.052922 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:10:55.052936 (+ 14us) server_negotiation.cc:407] Connection header received
0620 19:10:55.053065 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:10:55.053067 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:10:55.053122 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:10:55.053208 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:10:55.053487 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:55.054019 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:55.055060 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:10:55.055281 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:10:55.055434 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:10:55.055727 (+ 293us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:10:55.056188 (+ 461us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:10:55.056334 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 19:10:55.056427 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 19:10:55.056487 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 19:10:55.062947 20216 raft_consensus.cc:1261] T fca4a269b5af4fcd92bcedc7630a424d P 8ff79f8bd6ab47f9bfbc8fadd34fbb00 [term 1 FOLLOWER]: Refusing update from remote peer 98a1172a78e34632ad35688f1192a6af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:10:55.062945 20484 raft_consensus.cc:1261] T fca4a269b5af4fcd92bcedc7630a424d P 126d56d590ae455cb0651a29df830c7a [term 1 FOLLOWER]: Refusing update from remote peer 98a1172a78e34632ad35688f1192a6af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:10:55.063709 20555 consensus_queue.cc:1035] T fca4a269b5af4fcd92bcedc7630a424d P 98a1172a78e34632ad35688f1192a6af [LEADER]: Connected to new peer: Peer: permanent_uuid: "126d56d590ae455cb0651a29df830c7a" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 45815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:10:55.063915 20557 consensus_queue.cc:1035] T fca4a269b5af4fcd92bcedc7630a424d P 98a1172a78e34632ad35688f1192a6af [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ff79f8bd6ab47f9bfbc8fadd34fbb00" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 37433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:10:55.072455 20560 mvcc.cc:205] Tried to move back new op lower bound from 7299000955130155008 to 7299000955046436864. Current Snapshot: MvccSnapshot[applied={T|T < 7299000955130155008}]
I20260620 19:10:55.073239 20561 mvcc.cc:205] Tried to move back new op lower bound from 7299000955130155008 to 7299000955046436864. Current Snapshot: MvccSnapshot[applied={T|T < 7299000955130155008}]
I20260620 19:10:55.082782 20071 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43580:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:10:55.082990 20071 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43580:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:10:55.085909 20071 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3ced55e2ed6b48a6853b70f5a8a4ea2b: Sending DeleteTablet for 3 replicas of tablet fca4a269b5af4fcd92bcedc7630a424d
I20260620 19:10:55.086881 20464 tablet_service.cc:1514] Processing DeleteTablet for tablet fca4a269b5af4fcd92bcedc7630a424d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:10:55 UTC) from {username='kudu', principal='kudu/127.9.155.126@KRBTEST.COM'} at 127.0.0.1:42156
I20260620 19:10:55.087035 20330 tablet_service.cc:1514] Processing DeleteTablet for tablet fca4a269b5af4fcd92bcedc7630a424d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:10:55 UTC) from {username='kudu', principal='kudu/127.9.155.126@KRBTEST.COM'} at 127.0.0.1:56084
I20260620 19:10:55.087483 20196 tablet_service.cc:1514] Processing DeleteTablet for tablet fca4a269b5af4fcd92bcedc7630a424d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:10:55 UTC) from {username='kudu', principal='kudu/127.9.155.126@KRBTEST.COM'} at 127.0.0.1:52758
I20260620 19:10:55.087926 20572 tablet_replica.cc:330] T fca4a269b5af4fcd92bcedc7630a424d P 8ff79f8bd6ab47f9bfbc8fadd34fbb00: stopping tablet replica
I20260620 19:10:55.088322 20571 tablet_replica.cc:330] T fca4a269b5af4fcd92bcedc7630a424d P 126d56d590ae455cb0651a29df830c7a: stopping tablet replica
I20260620 19:10:55.088332 20572 raft_consensus.cc:2229] T fca4a269b5af4fcd92bcedc7630a424d P 8ff79f8bd6ab47f9bfbc8fadd34fbb00 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:10:55.088609 20571 raft_consensus.cc:2229] T fca4a269b5af4fcd92bcedc7630a424d P 126d56d590ae455cb0651a29df830c7a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:10:55.088666 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 20133
I20260620 19:10:55.088706 20572 raft_consensus.cc:2258] T fca4a269b5af4fcd92bcedc7630a424d P 8ff79f8bd6ab47f9bfbc8fadd34fbb00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:10:55.088867 20571 raft_consensus.cc:2258] T fca4a269b5af4fcd92bcedc7630a424d P 126d56d590ae455cb0651a29df830c7a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:10:55.090623 20571 ts_tablet_manager.cc:1905] T fca4a269b5af4fcd92bcedc7630a424d P 126d56d590ae455cb0651a29df830c7a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:10:55.092315 20570 tablet_replica.cc:330] T fca4a269b5af4fcd92bcedc7630a424d P 98a1172a78e34632ad35688f1192a6af: stopping tablet replica
I20260620 19:10:55.092656 20570 raft_consensus.cc:2229] T fca4a269b5af4fcd92bcedc7630a424d P 98a1172a78e34632ad35688f1192a6af [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:10:55.093099 20570 raft_consensus.cc:2258] T fca4a269b5af4fcd92bcedc7630a424d P 98a1172a78e34632ad35688f1192a6af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:10:55.093329 20571 ts_tablet_manager.cc:1918] T fca4a269b5af4fcd92bcedc7630a424d P 126d56d590ae455cb0651a29df830c7a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:10:55.093436 20571 log.cc:1196] T fca4a269b5af4fcd92bcedc7630a424d P 126d56d590ae455cb0651a29df830c7a: Deleting WAL directory at /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/minicluster-data/ts-2/wal/wals/fca4a269b5af4fcd92bcedc7630a424d
I20260620 19:10:55.093750 20571 ts_tablet_manager.cc:1939] T fca4a269b5af4fcd92bcedc7630a424d P 126d56d590ae455cb0651a29df830c7a: Deleting consensus metadata
I20260620 19:10:55.094636 20570 ts_tablet_manager.cc:1905] T fca4a269b5af4fcd92bcedc7630a424d P 98a1172a78e34632ad35688f1192a6af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:10:55.095103 20055 catalog_manager.cc:4841] TS 126d56d590ae455cb0651a29df830c7a (127.9.155.67:45815): tablet fca4a269b5af4fcd92bcedc7630a424d (table test-table [id=3c494a01444642c18ae3282ad2ea558e]) successfully deleted
I20260620 19:10:55.096975 20570 ts_tablet_manager.cc:1918] T fca4a269b5af4fcd92bcedc7630a424d P 98a1172a78e34632ad35688f1192a6af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:10:55.097124 20570 log.cc:1196] T fca4a269b5af4fcd92bcedc7630a424d P 98a1172a78e34632ad35688f1192a6af: Deleting WAL directory at /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982595231230-9837-0/minicluster-data/ts-1/wal/wals/fca4a269b5af4fcd92bcedc7630a424d
I20260620 19:10:55.097396 20570 ts_tablet_manager.cc:1939] T fca4a269b5af4fcd92bcedc7630a424d P 98a1172a78e34632ad35688f1192a6af: Deleting consensus metadata
W20260620 19:10:55.098452 20058 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.155.65:37433 (error 108)
I20260620 19:10:55.098507 20055 catalog_manager.cc:4841] TS 98a1172a78e34632ad35688f1192a6af (127.9.155.66:40949): tablet fca4a269b5af4fcd92bcedc7630a424d (table test-table [id=3c494a01444642c18ae3282ad2ea558e]) successfully deleted
I20260620 19:10:55.099287 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 20267
I20260620 19:10:55.099462 20545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:10:55.099165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.65:37433 (local address 127.0.0.1:52772)
0620 19:10:55.099307 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 19:10:55.099381 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.65:37433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 19:10:55.100991 20058 catalog_manager.cc:4584] TS 8ff79f8bd6ab47f9bfbc8fadd34fbb00 (127.9.155.65:37433): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fca4a269b5af4fcd92bcedc7630a424d: Network error: Client connection negotiation failed: client connection to 127.9.155.65:37433: connect: Connection refused (error 111)
I20260620 19:10:55.104789 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 20401
I20260620 19:10:55.109616 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 20039
2026-06-20T19:10:55Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2232 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:10:55 dist-test-slave-n3fx krb5kdc[20576](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:10:55 dist-test-slave-n3fx krb5kdc[20576](info): set up 2 sockets
Jun 20 19:10:55 dist-test-slave-n3fx krb5kdc[20576](info): commencing operation
krb5kdc: starting...
W20260620 19:10:56.081009 9837 mini_kdc.cc:121] Time spent starting KDC: real 0.951s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/krb5kdc/test-user.keytab.
Jun 20 19:10:56 dist-test-slave-n3fx krb5kdc[20576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:10:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:10:56Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.9.155.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.9.155.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.9.155.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/krb5kdc/rangeradmin_127.9.155.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.9.155.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/krb5kdc/rangeradmin_127.9.155.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.9.155.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.9.155.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.9.155.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/krb5kdc/rangerlookup_127.9.155.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.9.155.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/krb5kdc/rangerlookup_127.9.155.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.9.155.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.155.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.155.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/krb5kdc/HTTP_127.9.155.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.9.155.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/krb5kdc/HTTP_127.9.155.84@KRBTEST.COM.keytab.
I20260620 19:10:56.244688 9837 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task74hYKn/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/minicluster-data/postgres -l logfile start
2026-06-20 19:10:58.703 UTC [20606] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 19:10:58.703 UTC [20606] LOG: listening on IPv4 address "127.9.155.84", port 53889
2026-06-20 19:10:58.707 UTC [20606] LOG: listening on Unix socket "/tmp/.s.PGSQL.53889"
2026-06-20 19:10:58.733 UTC [20609] LOG: database system was shut down at 2026-06-20 19:10:57 UTC
2026-06-20 19:10:58.737 UTC [20606] LOG: database system is ready to accept connections
I20260620 19:10:59.608757 9837 mini_postgres.cc:96] Postgres bound to 53889
2026-06-20 19:10:59.614 UTC [20617] FATAL: database "slave" does not exist
127.9.155.84:53889 - accepting connections
I20260620 19:10:59.615398 9837 mini_ranger.cc:161] Starting Ranger...
I20260620 19:10:59.630168 9837 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 19:11:00.146030 9837 mini_ranger.cc:87] Created ranger Postgres database
I20260620 19:11:00.146124 9837 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/minicluster-data/ranger-admin
2026-06-20 19:11:00,419 [I] DB FLAVOR :POSTGRES
2026-06-20 19:11:00,419 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 19:11:00,419 [I] Checking connection..
2026-06-20 19:11:00,419 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task74hYKn/build/debug/bin/postgresql.jar:/tmp/dist-test-task74hYKn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.155.84:53889/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 19:11:00,667 [I] Checking connection passed.
2026-06-20 19:11:00,667 [I] --------- Verifying version history table ---------
2026-06-20 19:11:00,667 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task74hYKn/build/debug/bin/postgresql.jar:/tmp/dist-test-task74hYKn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.155.84:53889/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 19:11:00,909 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 19:11:00,910 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task74hYKn/build/debug/bin/postgresql.jar:/tmp/dist-test-task74hYKn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.155.84:53889/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 19:11:01,169 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 19:11:01,170 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 19:11:01,170 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task74hYKn/build/debug/bin/postgresql.jar:/tmp/dist-test-task74hYKn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.155.84:53889/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task74hYKn/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 19:11:01.417 UTC [20681] WARNING: there is no transaction in progress
2026-06-20 19:11:01,422 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 19:11:01,423 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task74hYKn/build/debug/bin/postgresql.jar:/tmp/dist-test-task74hYKn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.155.84:53889/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 19:11:01,670 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 19:11:01,670 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 19:11:01,671 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task74hYKn/build/debug/bin/postgresql.jar:/tmp/dist-test-task74hYKn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.155.84:53889/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 19:11:01,919 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task74hYKn/build/debug/bin/postgresql.jar:/tmp/dist-test-task74hYKn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.155.84:53889/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 19:11:02,178 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task74hYKn/build/debug/bin/postgresql.jar:/tmp/dist-test-task74hYKn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.155.84:53889/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-n3fx.c.gcp-upstream.internal','N') ;"
2026-06-20 19:11:02,425 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task74hYKn/build/debug/bin/postgresql.jar:/tmp/dist-test-task74hYKn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.155.84:53889/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 19:11:02,684 [I] Table x_portal_user does not exist in database ranger
2026-06-20 19:11:02,684 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task74hYKn/build/debug/bin/postgresql.jar:/tmp/dist-test-task74hYKn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.155.84:53889/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 19:11:02,919 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 19:11:02,919 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task74hYKn/build/debug/bin/postgresql.jar:/tmp/dist-test-task74hYKn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.155.84:53889/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 19:11:03,162 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task74hYKn/build/debug/bin/postgresql.jar:/tmp/dist-test-task74hYKn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.155.84:53889/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 19:11:03,424 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 19:11:03,424 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task74hYKn/build/debug/bin/postgresql.jar:/tmp/dist-test-task74hYKn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.155.84:53889/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task74hYKn/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 19:11:04.432 UTC [20789] WARNING: there is no transaction in progress
2026-06-20 19:11:04.449 UTC [20789] WARNING: there is no transaction in progress
2026-06-20 19:11:04.467 UTC [20789] WARNING: there is no transaction in progress
2026-06-20 19:11:04.482 UTC [20789] WARNING: there is no transaction in progress
2026-06-20 19:11:04.499 UTC [20789] WARNING: there is no transaction in progress
2026-06-20 19:11:04.518 UTC [20789] WARNING: there is no transaction in progress
2026-06-20 19:11:04.576 UTC [20789] WARNING: there is no transaction in progress
2026-06-20 19:11:04.583 UTC [20789] WARNING: there is no transaction in progress
2026-06-20 19:11:04.598 UTC [20789] WARNING: there is no transaction in progress
2026-06-20 19:11:04.607 UTC [20789] WARNING: there is no transaction in progress
2026-06-20 19:11:04.618 UTC [20789] WARNING: there is no transaction in progress
2026-06-20 19:11:04.633 UTC [20789] WARNING: there is no transaction in progress
2026-06-20 19:11:04.642 UTC [20789] WARNING: there is no transaction in progress
2026-06-20 19:11:04.649 UTC [20789] WARNING: there is no transaction in progress
2026-06-20 19:11:04.655 UTC [20789] WARNING: there is no transaction in progress
2026-06-20 19:11:04.661 UTC [20789] WARNING: there is no transaction in progress
2026-06-20 19:11:05,328 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 19:11:05,328 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task74hYKn/build/debug/bin/postgresql.jar:/tmp/dist-test-task74hYKn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.155.84:53889/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 19:11:05,556 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 19:11:05,556 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task74hYKn/build/debug/bin/postgresql.jar:/tmp/dist-test-task74hYKn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.155.84:53889/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 19:11:05,817 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 19:11:05,817 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task74hYKn/build/debug/bin/postgresql.jar:/tmp/dist-test-task74hYKn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.155.84:53889/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 19:11:06,081 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 19:11:06,082 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task74hYKn/build/debug/bin/postgresql.jar:/tmp/dist-test-task74hYKn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.155.84:53889/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 19:11:06,314 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task74hYKn/build/debug/bin/postgresql.jar:/tmp/dist-test-task74hYKn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.155.84:53889/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 19:11:06,576 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task74hYKn/build/debug/bin/postgresql.jar:/tmp/dist-test-task74hYKn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.155.84:53889/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-n3fx.c.gcp-upstream.internal';"
2026-06-20 19:11:06,820 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 19:11:06,821 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task74hYKn/build/debug/bin/postgresql.jar:/tmp/dist-test-task74hYKn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.155.84:53889/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 19:11:07,079 [I] DB_PATCHES have already been applied
I20260620 19:11:07.084649 9837 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/minicluster-data/ranger-admin:/tmp/dist-test-task74hYKn/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task74hYKn/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task74hYKn/build/debug/bin/postgresql.jar:/tmp/dist-test-task74hYKn/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 19:11:07.084713 9837 mini_ranger.cc:193] Using host: 127.9.155.84
I20260620 19:11:07.088665 9837 mini_ranger.cc:227] Ranger admin URL: http://127.9.155.84:37201
Jun 20, 2026 7:11:07 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 7:11:07 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 7:11:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 7:11:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 7:11:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 7:11:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 7:11:07 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 19:11:08.071762 9837 mini_ranger.cc:160] Time spent starting Ranger: real 8.456s user 0.000s sys 0.010s
Jun 20, 2026 7:11:28 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 7:11:28 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Jun 20, 2026 7:11:28 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 7:11:28 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 7:11:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 7:11:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 7:11:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 7:11:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 7:11:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 7:11:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 7:11:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 7:11:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 7:11:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 7:11:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 7:11:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 7:11:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 7:11:28 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 7:11:29 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 19:11:30.008790 9837 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.9.155.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.9.155.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.9.155.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/krb5kdc/rangerkms_127.9.155.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.9.155.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/krb5kdc/rangerkms_127.9.155.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.9.155.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.9.155.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.155.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/krb5kdc/HTTP_127.9.155.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.9.155.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/krb5kdc/HTTP_127.9.155.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 19:11:30 dist-test-slave-n3fx krb5kdc[20576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982690, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 19:11:30.098109 9837 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 19:11:30.107874 9837 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 19:11:31.000703 9837 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 19:11:31.000792 9837 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/minicluster-data/ranger-kms
I20260620 19:11:31.000882 9837 mini_ranger_kms.cc:227] Using postgres at 127.9.155.84:53889
2026-06-20 19:11:31,165 [I] DB FLAVOR :POSTGRES
2026-06-20 19:11:31,165 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 19:11:31,165 [I] Checking connection
2026-06-20 19:11:31,165 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task74hYKn/build/debug/bin/postgresql.jar:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.155.84:53889/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 19:11:31,427 [I] connection success
2026-06-20 19:11:31,428 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 19:11:31,428 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 19:11:31,428 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task74hYKn/build/debug/bin/postgresql.jar:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.155.84:53889/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 19:11:31,665 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 19:11:31,665 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 19:11:31,665 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 19:11:31,665 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task74hYKn/build/debug/bin/postgresql.jar:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.155.84:53889/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 19:11:31,921 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 19:11:32.387755 9837 mini_ranger_kms.cc:310] Created kms service
I20260620 19:11:32.569590 9837 mini_ranger_kms.cc:326] Created kudu user
I20260620 19:11:32.613106 9837 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 19:11:32 dist-test-slave-n3fx krb5kdc[20576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982690, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.9.155.84@KRBTEST.COM
I20260620 19:11:33.033308 9837 mini_ranger_kms.cc:384] Added ranger policy
I20260620 19:11:33.033464 9837 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/minicluster-data/ranger-kms:/tmp/dist-test-task74hYKn/build/debug/bin/postgresql.jar:/tmp/dist-test-task74hYKn/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task74hYKn/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 19:11:33.033485 9837 mini_ranger_kms.cc:240] Using host: 127.9.155.84
I20260620 19:11:33.037451 9837 mini_ranger_kms.cc:276] Ranger KMS PID: 20969
I20260620 19:11:33.037581 9837 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.9.155.84:47653
Jun 20, 2026 7:11:34 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 7:11:34 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 7:11:34 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 7:11:34 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 7:11:34 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 7:11:34 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 19:11:35.120361 9837 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.022s user 0.000s sys 0.148s
I20260620 19:11:35.120493 9837 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 19:11:35.120601 9837 mini_ranger_kms.cc:401] 127.9.155.84:47653/kms/v1/keys
Jun 20 19:11:35 dist-test-slave-n3fx krb5kdc[20576](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781982695, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.155.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 19:11:35 dist-test-slave-n3fx krb5kdc[20576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982695, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.155.84@KRBTEST.COM for HTTP/127.9.155.84@KRBTEST.COM
Jun 20 19:11:36 dist-test-slave-n3fx krb5kdc[20576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982695, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.155.84@KRBTEST.COM for HTTP/127.9.155.84@KRBTEST.COM
Jun 20 19:11:38 dist-test-slave-n3fx krb5kdc[20576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982698, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.9.155.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.155.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.155.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab.
Entry for principal kudu/127.9.155.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab.
I20260620 19:11:38.472309 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:42645
--webserver_interface=127.9.155.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab
--principal=kudu/127.9.155.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:41387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.155.126:42645
--ranger_config_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.155.84:47653/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:11:38.585541 21020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:11:38.585852 21020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:11:38.585911 21020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:11:38.590005 21020 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:11:38.590071 21020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:11:38.590097 21020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:11:38.590119 21020 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:11:38.590154 21020 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:11:38.595642 21020 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:41387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/minicluster-data/master-0/wal
--ranger_kms_url=127.9.155.84:47653/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.9.155.126:42645
--ranger_config_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab
--principal=kudu/127.9.155.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:42645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.155.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21020
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:11:38.596784 21020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:11:38.597805 21020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:11:38.605844 21026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:11:38.605858 21025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:11:38.605851 21028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:11:38.606287 21020 server_base.cc:1034] running on GCE node
I20260620 19:11:38.606916 21020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:11:38.608348 21020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:11:38.609526 21020 hybrid_clock.cc:648] HybridClock initialized: now 1781982698609501 us; error 38 us; skew 500 ppm
Jun 20 19:11:38 dist-test-slave-n3fx krb5kdc[20576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982698, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:11:38.618418 21020 init.cc:377] Logged in from keytab as kudu/127.9.155.126@KRBTEST.COM (short username kudu)
I20260620 19:11:38.619642 21020 webserver.cc:460] Webserver started at http://127.9.155.126:40051/ using document root <none> and password file <none>
I20260620 19:11:38.620334 21020 fs_manager.cc:359] Metadata directory not provided
I20260620 19:11:38.620409 21020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:11:38.620623 21020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 19:11:38 dist-test-slave-n3fx krb5kdc[20576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982698, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.126@KRBTEST.COM for HTTP/127.9.155.84@KRBTEST.COM
I20260620 19:11:38.676978 21020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/minicluster-data/master-0/data/instance:
uuid: "e2b28111327940e3a1579d1fe9ba6e3e"
format_stamp: "Formatted at 2026-06-20 19:11:38 on dist-test-slave-n3fx"
server_key: "ea8af09221541f6dc74b023cec49d400"
server_key_iv: "2922920d5dde0d8e8f8797315a3f34e4"
server_key_version: "kuduclusterkey@0"
I20260620 19:11:38.677825 21020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/minicluster-data/master-0/wal/instance:
uuid: "e2b28111327940e3a1579d1fe9ba6e3e"
format_stamp: "Formatted at 2026-06-20 19:11:38 on dist-test-slave-n3fx"
server_key: "ea8af09221541f6dc74b023cec49d400"
server_key_iv: "2922920d5dde0d8e8f8797315a3f34e4"
server_key_version: "kuduclusterkey@0"
I20260620 19:11:38.682807 21020 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 19:11:38.700203 21035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:38.701858 21020 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:11:38.702005 21020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/minicluster-data/master-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/minicluster-data/master-0/wal
uuid: "e2b28111327940e3a1579d1fe9ba6e3e"
format_stamp: "Formatted at 2026-06-20 19:11:38 on dist-test-slave-n3fx"
server_key: "ea8af09221541f6dc74b023cec49d400"
server_key_iv: "2922920d5dde0d8e8f8797315a3f34e4"
server_key_version: "kuduclusterkey@0"
I20260620 19:11:38.702153 21020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:11:38.736711 21020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:11:38.740100 21020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:11:38.740381 21020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:11:38.748495 21020 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.126:42645
I20260620 19:11:38.748503 21087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.126:42645 every 8 connection(s)
I20260620 19:11:38.749644 21020 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
I20260620 19:11:38.750413 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 21020
I20260620 19:11:38.750537 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/minicluster-data/master-0/wal/instance
I20260620 19:11:38.750860 9837 external_mini_cluster.cc:1428] Setting key c0a0dab80b7e3547ed612816c663fe2a
I20260620 19:11:38.753750 21088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 19:11:38 dist-test-slave-n3fx krb5kdc[20576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982698, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:11:38.761224 21088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2b28111327940e3a1579d1fe9ba6e3e: Bootstrap starting.
I20260620 19:11:38.763855 21091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:38.752297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57954 (local address 127.9.155.126:42645)
0620 19:11:38.752711 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:38.752727 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:38.752761 (+ 34us) server_negotiation.cc:407] Connection header received
0620 19:11:38.753385 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:38.753402 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:38.753733 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:38.754054 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:11:38.754813 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:38.755794 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:38.756676 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:38.757152 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:38.757303 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:38.759614 (+ 2311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:11:38.759619 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:11:38.759631 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:11:38.759658 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:11:38.761136 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:11:38.761586 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:11:38.761591 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:11:38.761595 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:11:38.761700 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:11:38.761947 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:11:38.761950 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:11:38.761952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:11:38.762372 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:11:38.762550 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:38.762837 (+ 287us) server_negotiation.cc:299] Negotiation successful
0620 19:11:38.763122 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":115,"threads_started":1}
I20260620 19:11:38.763908 21088 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e2b28111327940e3a1579d1fe9ba6e3e: Neither blocks nor log segments found. Creating new log.
I20260620 19:11:38.765214 21088 log.cc:824] T 00000000000000000000000000000000 P e2b28111327940e3a1579d1fe9ba6e3e: Log is configured to *not* fsync() on all Append() calls
I20260620 19:11:38.767561 21088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2b28111327940e3a1579d1fe9ba6e3e: No bootstrap required, opened a new log
I20260620 19:11:38.771477 21088 raft_consensus.cc:348] T 00000000000000000000000000000000 P e2b28111327940e3a1579d1fe9ba6e3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2b28111327940e3a1579d1fe9ba6e3e" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 42645 } }
I20260620 19:11:38.771780 21088 raft_consensus.cc:374] T 00000000000000000000000000000000 P e2b28111327940e3a1579d1fe9ba6e3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:11:38.771836 21088 raft_consensus.cc:729] T 00000000000000000000000000000000 P e2b28111327940e3a1579d1fe9ba6e3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2b28111327940e3a1579d1fe9ba6e3e, State: Initialized, Role: FOLLOWER
I20260620 19:11:38.772332 21088 consensus_queue.cc:260] T 00000000000000000000000000000000 P e2b28111327940e3a1579d1fe9ba6e3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2b28111327940e3a1579d1fe9ba6e3e" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 42645 } }
I20260620 19:11:38.772485 21088 raft_consensus.cc:388] T 00000000000000000000000000000000 P e2b28111327940e3a1579d1fe9ba6e3e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:11:38.772544 21088 raft_consensus.cc:482] T 00000000000000000000000000000000 P e2b28111327940e3a1579d1fe9ba6e3e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:11:38.772635 21088 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e2b28111327940e3a1579d1fe9ba6e3e [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:11:38.773761 21088 raft_consensus.cc:504] T 00000000000000000000000000000000 P e2b28111327940e3a1579d1fe9ba6e3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2b28111327940e3a1579d1fe9ba6e3e" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 42645 } }
I20260620 19:11:38.774118 21088 leader_election.cc:302] T 00000000000000000000000000000000 P e2b28111327940e3a1579d1fe9ba6e3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e2b28111327940e3a1579d1fe9ba6e3e; no voters:
I20260620 19:11:38.774466 21088 leader_election.cc:288] T 00000000000000000000000000000000 P e2b28111327940e3a1579d1fe9ba6e3e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:11:38.774560 21093 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e2b28111327940e3a1579d1fe9ba6e3e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:11:38.774865 21093 raft_consensus.cc:686] T 00000000000000000000000000000000 P e2b28111327940e3a1579d1fe9ba6e3e [term 1 LEADER]: Becoming Leader. State: Replica: e2b28111327940e3a1579d1fe9ba6e3e, State: Running, Role: LEADER
I20260620 19:11:38.775259 21093 consensus_queue.cc:237] T 00000000000000000000000000000000 P e2b28111327940e3a1579d1fe9ba6e3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2b28111327940e3a1579d1fe9ba6e3e" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 42645 } }
I20260620 19:11:38.775763 21088 sys_catalog.cc:564] T 00000000000000000000000000000000 P e2b28111327940e3a1579d1fe9ba6e3e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:11:38.776741 21095 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2b28111327940e3a1579d1fe9ba6e3e [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2b28111327940e3a1579d1fe9ba6e3e. Latest consensus state: current_term: 1 leader_uuid: "e2b28111327940e3a1579d1fe9ba6e3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2b28111327940e3a1579d1fe9ba6e3e" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 42645 } } }
I20260620 19:11:38.776857 21095 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2b28111327940e3a1579d1fe9ba6e3e [sys.catalog]: This master's current role is: LEADER
I20260620 19:11:38.777180 21094 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2b28111327940e3a1579d1fe9ba6e3e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e2b28111327940e3a1579d1fe9ba6e3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2b28111327940e3a1579d1fe9ba6e3e" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 42645 } } }
I20260620 19:11:38.777266 21094 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2b28111327940e3a1579d1fe9ba6e3e [sys.catalog]: This master's current role is: LEADER
I20260620 19:11:38.777531 21102 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:11:38.777848 21088 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 19:11:38.784029 21102 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:11:38.792752 21102 catalog_manager.cc:1347] Generated new cluster ID: 5bc9b7f652204bcab18d2f1785b99f66
I20260620 19:11:38.792834 21102 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:11:38.804898 21102 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:11:38.806162 21102 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:11:38.817521 21102 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e2b28111327940e3a1579d1fe9ba6e3e: Generated new TSK 0
I20260620 19:11:38.818413 21102 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.155.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.155.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.155.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.65.keytab.
Entry for principal kudu/127.9.155.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.65.keytab.
I20260620 19:11:39.560932 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.155.84:47653/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.65:0
--local_ip_for_outbound_sockets=127.9.155.65
--webserver_interface=127.9.155.65
--webserver_port=0
--tserver_master_addrs=127.9.155.126:42645
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.65.keytab
--principal=kudu/127.9.155.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:41387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:11:39.707571 21133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:11:39.707855 21133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:11:39.707901 21133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:11:39.712543 21133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:11:39.712634 21133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:11:39.712738 21133 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.65
I20260620 19:11:39.720800 21133 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:41387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.9.155.84:47653/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.65.keytab
--principal=kudu/127.9.155.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.155.65
--webserver_port=0
--tserver_master_addrs=127.9.155.126:42645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21133
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.65
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:11:39.722168 21133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:11:39.723277 21133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:11:39.734724 21141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:11:39.734867 21143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:11:39.735129 21140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:11:39.735224 21133 server_base.cc:1034] running on GCE node
I20260620 19:11:39.735787 21133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:11:39.736616 21133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:11:39.737812 21133 hybrid_clock.cc:648] HybridClock initialized: now 1781982699737785 us; error 43 us; skew 500 ppm
Jun 20 19:11:39 dist-test-slave-n3fx krb5kdc[20576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982699, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:11:39.741451 21133 init.cc:377] Logged in from keytab as kudu/127.9.155.65@KRBTEST.COM (short username kudu)
I20260620 19:11:39.742635 21133 webserver.cc:460] Webserver started at http://127.9.155.65:35193/ using document root <none> and password file <none>
I20260620 19:11:39.743378 21133 fs_manager.cc:359] Metadata directory not provided
I20260620 19:11:39.743479 21133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:11:39.743750 21133 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 19:11:39 dist-test-slave-n3fx krb5kdc[20576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982699, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.65@KRBTEST.COM for HTTP/127.9.155.84@KRBTEST.COM
I20260620 19:11:39.763259 21133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/minicluster-data/ts-0/data/instance:
uuid: "49652bd0070844ceba74158fb694a66d"
format_stamp: "Formatted at 2026-06-20 19:11:39 on dist-test-slave-n3fx"
server_key: "df0d232bf036c41c335dfefbbb4bc094"
server_key_iv: "11bd3696f3f30ac5506430aec87d1ab4"
server_key_version: "kuduclusterkey@0"
I20260620 19:11:39.763878 21133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/minicluster-data/ts-0/wal/instance:
uuid: "49652bd0070844ceba74158fb694a66d"
format_stamp: "Formatted at 2026-06-20 19:11:39 on dist-test-slave-n3fx"
server_key: "df0d232bf036c41c335dfefbbb4bc094"
server_key_iv: "11bd3696f3f30ac5506430aec87d1ab4"
server_key_version: "kuduclusterkey@0"
I20260620 19:11:39.771661 21133 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260620 19:11:39.799912 21151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:39.801707 21133 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.002s sys 0.002s
I20260620 19:11:39.801841 21133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/minicluster-data/ts-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/minicluster-data/ts-0/wal
uuid: "49652bd0070844ceba74158fb694a66d"
format_stamp: "Formatted at 2026-06-20 19:11:39 on dist-test-slave-n3fx"
server_key: "df0d232bf036c41c335dfefbbb4bc094"
server_key_iv: "11bd3696f3f30ac5506430aec87d1ab4"
server_key_version: "kuduclusterkey@0"
I20260620 19:11:39.801978 21133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:11:39.823721 21133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:11:39.827482 21133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:11:39.827695 21133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:11:39.828430 21133 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:11:39.829492 21133 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:11:39.829555 21133 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:39.829612 21133 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:11:39.829635 21133 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:39.840310 21265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.65:35935 every 8 connection(s)
I20260620 19:11:39.841389 21133 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.65:35935
I20260620 19:11:39.842628 21133 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
I20260620 19:11:39.849028 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 21133
I20260620 19:11:39.849165 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/minicluster-data/ts-0/wal/instance
I20260620 19:11:39.849445 9837 external_mini_cluster.cc:1428] Setting key f5270901da1cee361977d4d19161eabe
Jun 20 19:11:39 dist-test-slave-n3fx krb5kdc[20576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982699, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.65@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:11:39.859545 21268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:39.844981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.65:33915 (local address 127.9.155.126:42645)
0620 19:11:39.845276 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:39.845281 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:39.846277 (+ 996us) server_negotiation.cc:407] Connection header received
0620 19:11:39.847537 (+ 1260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:39.847541 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:39.847598 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:39.847667 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:11:39.849301 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:39.850105 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:39.851799 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:39.852152 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:39.852248 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:39.855218 (+ 2970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:11:39.855222 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:11:39.855225 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:11:39.855253 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:11:39.857114 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:11:39.857725 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:11:39.857728 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:11:39.857730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:11:39.857819 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:11:39.858228 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:11:39.858231 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:11:39.858233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:11:39.858458 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:11:39.858527 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:39.859223 (+ 696us) server_negotiation.cc:299] Negotiation successful
0620 19:11:39.859357 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":97,"threads_started":1}
I20260620 19:11:39.861171 21269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:39.845529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:42645 (local address 127.9.155.65:33915)
0620 19:11:39.846106 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:39.846164 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 19:11:39.847293 (+ 1129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:11:39.848078 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:11:39.848087 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:11:39.848475 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:11:39.849137 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:39.849156 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:39.850254 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:39.850257 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:39.851650 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:39.851673 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:39.852449 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:39.852452 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:39.852665 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:39.853343 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:11:39.853363 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:11:39.855015 (+ 1652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:11:39.857238 (+ 2223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:11:39.857245 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:11:39.857264 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:11:39.857619 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:11:39.857924 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:11:39.857927 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:11:39.857929 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:11:39.858115 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:11:39.858607 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:11:39.858614 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:11:39.858946 (+ 332us) client_negotiation.cc:769] Sending connection context
0620 19:11:39.859784 (+ 838us) client_negotiation.cc:240] Negotiation successful
0620 19:11:39.860125 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":160,"threads_started":1}
I20260620 19:11:39.862529 21266 heartbeater.cc:347] Connected to a master server at 127.9.155.126:42645
I20260620 19:11:39.862787 21266 heartbeater.cc:464] Registering TS with master...
I20260620 19:11:39.863371 21266 heartbeater.cc:511] Master 127.9.155.126:42645 requested a full tablet report, sending...
I20260620 19:11:39.865095 21052 ts_manager.cc:194] Registered new tserver with Master: 49652bd0070844ceba74158fb694a66d (127.9.155.65:35935)
I20260620 19:11:39.866590 21052 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.155.65@KRBTEST.COM'} at 127.9.155.65:33915
WARNING: no policy specified for kudu/127.9.155.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.155.66@KRBTEST.COM" created.
Jun 20 19:11:39 dist-test-slave-n3fx krb5kdc[20576](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781982699, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.155.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.66.keytab.
Entry for principal kudu/127.9.155.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.66.keytab.
I20260620 19:11:39.887212 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.155.84:47653/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.66:0
--local_ip_for_outbound_sockets=127.9.155.66
--webserver_interface=127.9.155.66
--webserver_port=0
--tserver_master_addrs=127.9.155.126:42645
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.66.keytab
--principal=kudu/127.9.155.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:41387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:11:40.009069 21272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:11:40.009390 21272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:11:40.009454 21272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:11:40.013793 21272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:11:40.013882 21272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:11:40.013969 21272 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.66
I20260620 19:11:40.021543 21272 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:41387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.9.155.84:47653/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.66.keytab
--principal=kudu/127.9.155.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.155.66
--webserver_port=0
--tserver_master_addrs=127.9.155.126:42645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21272
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.66
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:11:40.023070 21272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:11:40.024304 21272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:11:40.035463 21281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:11:40.035547 21279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:11:40.035463 21278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:11:40.036578 21272 server_base.cc:1034] running on GCE node
I20260620 19:11:40.037101 21272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:11:40.037757 21272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:11:40.038914 21272 hybrid_clock.cc:648] HybridClock initialized: now 1781982700038883 us; error 49 us; skew 500 ppm
Jun 20 19:11:40 dist-test-slave-n3fx krb5kdc[20576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982700, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:11:40.042557 21272 init.cc:377] Logged in from keytab as kudu/127.9.155.66@KRBTEST.COM (short username kudu)
I20260620 19:11:40.043540 21272 webserver.cc:460] Webserver started at http://127.9.155.66:33051/ using document root <none> and password file <none>
I20260620 19:11:40.044131 21272 fs_manager.cc:359] Metadata directory not provided
I20260620 19:11:40.044245 21272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:11:40.044469 21272 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 19:11:40 dist-test-slave-n3fx krb5kdc[20576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982700, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.66@KRBTEST.COM for HTTP/127.9.155.84@KRBTEST.COM
I20260620 19:11:40.072034 21272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/minicluster-data/ts-1/data/instance:
uuid: "22fbb8de4a52429fbc4f887f161322d4"
format_stamp: "Formatted at 2026-06-20 19:11:40 on dist-test-slave-n3fx"
server_key: "db569691630a99780a68391abad62b3a"
server_key_iv: "5e77b631d40ddbf86ac60013ed67eaab"
server_key_version: "kuduclusterkey@0"
I20260620 19:11:40.072784 21272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/minicluster-data/ts-1/wal/instance:
uuid: "22fbb8de4a52429fbc4f887f161322d4"
format_stamp: "Formatted at 2026-06-20 19:11:40 on dist-test-slave-n3fx"
server_key: "db569691630a99780a68391abad62b3a"
server_key_iv: "5e77b631d40ddbf86ac60013ed67eaab"
server_key_version: "kuduclusterkey@0"
I20260620 19:11:40.080941 21272 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.003s sys 0.000s
I20260620 19:11:40.104480 21288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:40.106330 21272 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.002s sys 0.000s
I20260620 19:11:40.106487 21272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/minicluster-data/ts-1/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/minicluster-data/ts-1/wal
uuid: "22fbb8de4a52429fbc4f887f161322d4"
format_stamp: "Formatted at 2026-06-20 19:11:40 on dist-test-slave-n3fx"
server_key: "db569691630a99780a68391abad62b3a"
server_key_iv: "5e77b631d40ddbf86ac60013ed67eaab"
server_key_version: "kuduclusterkey@0"
I20260620 19:11:40.106642 21272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:11:40.145951 21272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:11:40.149022 21272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:11:40.149204 21272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:11:40.149819 21272 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:11:40.150655 21272 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:11:40.150702 21272 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:40.150755 21272 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:11:40.150784 21272 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:40.160862 21272 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.66:39907
I20260620 19:11:40.160931 21401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.66:39907 every 8 connection(s)
I20260620 19:11:40.161775 21272 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
I20260620 19:11:40.168746 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 21272
I20260620 19:11:40.168887 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/minicluster-data/ts-1/wal/instance
I20260620 19:11:40.169152 9837 external_mini_cluster.cc:1428] Setting key f17cbcbb4920b3522042133090fc0110
Jun 20 19:11:40 dist-test-slave-n3fx krb5kdc[20576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982700, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.66@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:11:40.182663 21268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:40.163643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.66:39121 (local address 127.9.155.126:42645)
0620 19:11:40.163790 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:40.163795 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:40.165001 (+ 1206us) server_negotiation.cc:407] Connection header received
0620 19:11:40.166259 (+ 1258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:40.166263 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:40.166308 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:40.166389 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:11:40.168022 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:40.168717 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:40.170516 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:40.170826 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:40.170933 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:40.173793 (+ 2860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:11:40.173797 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:11:40.173799 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:11:40.173827 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:11:40.178589 (+ 4762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:11:40.179863 (+ 1274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:11:40.179867 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:11:40.179869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:11:40.179964 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:11:40.181182 (+ 1218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:11:40.181186 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:11:40.181188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:11:40.181448 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:11:40.181525 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:40.182303 (+ 778us) server_negotiation.cc:299] Negotiation successful
0620 19:11:40.182453 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 19:11:40.184362 21404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:40.164053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:42645 (local address 127.9.155.66:39121)
0620 19:11:40.164777 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:40.164866 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0620 19:11:40.165995 (+ 1129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:11:40.166614 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:11:40.166625 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:11:40.167009 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:11:40.167859 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:40.167879 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:40.168859 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:40.168863 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:40.170348 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:40.170360 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:40.170989 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:40.170993 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:40.171205 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:40.171914 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:11:40.171942 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:11:40.173594 (+ 1652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:11:40.178752 (+ 5158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:11:40.178759 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:11:40.178780 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:11:40.179151 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:11:40.180085 (+ 934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:11:40.180088 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:11:40.180090 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:11:40.180318 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:11:40.181610 (+ 1292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:11:40.181617 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:11:40.181990 (+ 373us) client_negotiation.cc:769] Sending connection context
0620 19:11:40.182912 (+ 922us) client_negotiation.cc:240] Negotiation successful
0620 19:11:40.183282 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":188,"threads_started":1}
I20260620 19:11:40.185672 21402 heartbeater.cc:347] Connected to a master server at 127.9.155.126:42645
WARNING: no policy specified for kudu/127.9.155.67@KRBTEST.COM; defaulting to no policy
I20260620 19:11:40.185938 21402 heartbeater.cc:464] Registering TS with master...
I20260620 19:11:40.186590 21402 heartbeater.cc:511] Master 127.9.155.126:42645 requested a full tablet report, sending...
I20260620 19:11:40.187743 21052 ts_manager.cc:194] Registered new tserver with Master: 22fbb8de4a52429fbc4f887f161322d4 (127.9.155.66:39907)
I20260620 19:11:40.188503 21052 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.155.66@KRBTEST.COM'} at 127.9.155.66:39121
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.155.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.155.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.67.keytab.
Entry for principal kudu/127.9.155.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.67.keytab.
I20260620 19:11:40.207674 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.155.84:47653/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.67:0
--local_ip_for_outbound_sockets=127.9.155.67
--webserver_interface=127.9.155.67
--webserver_port=0
--tserver_master_addrs=127.9.155.126:42645
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.67.keytab
--principal=kudu/127.9.155.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:41387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 19:11:40 dist-test-slave-n3fx krb5kdc[20576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982699, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.126@KRBTEST.COM for HTTP/127.9.155.84@KRBTEST.COM
W20260620 19:11:40.347816 21407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:11:40.348181 21407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:11:40.348275 21407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:11:40.354274 21407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:11:40.354367 21407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:11:40.354451 21407 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.67
I20260620 19:11:40.362879 21407 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:41387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.9.155.84:47653/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.67.keytab
--principal=kudu/127.9.155.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.155.67
--webserver_port=0
--tserver_master_addrs=127.9.155.126:42645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21407
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.67
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:11:40.364363 21407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:11:40.365829 21407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:11:40.380884 21412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:11:40.381839 21413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:11:40.382053 21415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:11:40.382256 21407 server_base.cc:1034] running on GCE node
I20260620 19:11:40.382930 21407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:11:40.383811 21407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:11:40.387231 21407 hybrid_clock.cc:648] HybridClock initialized: now 1781982700387160 us; error 75 us; skew 500 ppm
Jun 20 19:11:40 dist-test-slave-n3fx krb5kdc[20576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982700, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:11:40.392036 21407 init.cc:377] Logged in from keytab as kudu/127.9.155.67@KRBTEST.COM (short username kudu)
I20260620 19:11:40.393342 21407 webserver.cc:460] Webserver started at http://127.9.155.67:36221/ using document root <none> and password file <none>
I20260620 19:11:40.394111 21407 fs_manager.cc:359] Metadata directory not provided
I20260620 19:11:40.394302 21407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:11:40.394585 21407 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 19:11:40 dist-test-slave-n3fx krb5kdc[20576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982700, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.67@KRBTEST.COM for HTTP/127.9.155.84@KRBTEST.COM
I20260620 19:11:40.427518 21407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/minicluster-data/ts-2/data/instance:
uuid: "5a43b5c5d38e4c85941380f50a90f1c6"
format_stamp: "Formatted at 2026-06-20 19:11:40 on dist-test-slave-n3fx"
server_key: "061b57112e9c0afe2e8715dadd74aa00"
server_key_iv: "bda90c13f883a8eefd7cc8854a1e69ac"
server_key_version: "kuduclusterkey@0"
I20260620 19:11:40.428319 21407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/minicluster-data/ts-2/wal/instance:
uuid: "5a43b5c5d38e4c85941380f50a90f1c6"
format_stamp: "Formatted at 2026-06-20 19:11:40 on dist-test-slave-n3fx"
server_key: "061b57112e9c0afe2e8715dadd74aa00"
server_key_iv: "bda90c13f883a8eefd7cc8854a1e69ac"
server_key_version: "kuduclusterkey@0"
I20260620 19:11:40.433441 21407 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260620 19:11:40.450064 21422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:40.451607 21407 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 19:11:40.451741 21407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/minicluster-data/ts-2/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/minicluster-data/ts-2/wal
uuid: "5a43b5c5d38e4c85941380f50a90f1c6"
format_stamp: "Formatted at 2026-06-20 19:11:40 on dist-test-slave-n3fx"
server_key: "061b57112e9c0afe2e8715dadd74aa00"
server_key_iv: "bda90c13f883a8eefd7cc8854a1e69ac"
server_key_version: "kuduclusterkey@0"
I20260620 19:11:40.451875 21407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:11:40.479840 21407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:11:40.483119 21407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:11:40.483333 21407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:11:40.484061 21407 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:11:40.485095 21407 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:11:40.485160 21407 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:40.485211 21407 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:11:40.485234 21407 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:11:40.499024 21407 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.67:37709
I20260620 19:11:40.499176 21535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.67:37709 every 8 connection(s)
I20260620 19:11:40.500120 21407 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
I20260620 19:11:40.507241 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 21407
I20260620 19:11:40.507360 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/minicluster-data/ts-2/wal/instance
I20260620 19:11:40.507691 9837 external_mini_cluster.cc:1428] Setting key 2c317d3b04b620d404ad3ff0f75e802a
Jun 20 19:11:40 dist-test-slave-n3fx krb5kdc[20576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982700, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.67@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:11:40.517737 21268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:40.502288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.67:53169 (local address 127.9.155.126:42645)
0620 19:11:40.502417 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:40.502422 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:40.503454 (+ 1032us) server_negotiation.cc:407] Connection header received
0620 19:11:40.504718 (+ 1264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:40.504722 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:40.504779 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:40.504869 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:11:40.506394 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:40.507254 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:40.510089 (+ 2835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:40.510446 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:40.510543 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:40.513502 (+ 2959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:11:40.513506 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:11:40.513509 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:11:40.513540 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:11:40.515185 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:11:40.515856 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:11:40.515860 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:11:40.515862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:11:40.515959 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:11:40.516393 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:11:40.516397 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:11:40.516399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:11:40.516651 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:11:40.516756 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:40.517416 (+ 660us) server_negotiation.cc:299] Negotiation successful
0620 19:11:40.517559 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 19:11:40.518745 21538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:40.502744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:42645 (local address 127.9.155.67:53169)
0620 19:11:40.503294 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:40.503351 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 19:11:40.504440 (+ 1089us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:11:40.505066 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:11:40.505076 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:11:40.505494 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:11:40.506240 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:40.506264 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:40.508202 (+ 1938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:40.508205 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:40.509899 (+ 1694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:40.509910 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:40.510606 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:40.510611 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:40.510809 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:40.511486 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:11:40.511507 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:11:40.513267 (+ 1760us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:11:40.515321 (+ 2054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:11:40.515328 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:11:40.515349 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:11:40.515729 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:11:40.516051 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:11:40.516054 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:11:40.516056 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:11:40.516286 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:11:40.516754 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:11:40.516762 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:11:40.517106 (+ 344us) client_negotiation.cc:769] Sending connection context
0620 19:11:40.517319 (+ 213us) client_negotiation.cc:240] Negotiation successful
0620 19:11:40.517684 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":134,"threads_started":1}
I20260620 19:11:40.519985 21536 heartbeater.cc:347] Connected to a master server at 127.9.155.126:42645
I20260620 19:11:40.520323 21536 heartbeater.cc:464] Registering TS with master...
I20260620 19:11:40.520933 21536 heartbeater.cc:511] Master 127.9.155.126:42645 requested a full tablet report, sending...
I20260620 19:11:40.522445 21052 ts_manager.cc:194] Registered new tserver with Master: 5a43b5c5d38e4c85941380f50a90f1c6 (127.9.155.67:37709)
I20260620 19:11:40.523209 21052 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.155.67@KRBTEST.COM'} at 127.9.155.67:53169
I20260620 19:11:40.535878 9837 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:11:40.547111 21268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:40.538018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57966 (local address 127.9.155.126:42645)
0620 19:11:40.538308 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:40.538313 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:40.538351 (+ 38us) server_negotiation.cc:407] Connection header received
0620 19:11:40.538526 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:40.538529 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:40.538583 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:40.538662 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:11:40.539497 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:40.540235 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:40.541349 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:40.541686 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:40.541792 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:40.543368 (+ 1576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:11:40.543372 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:11:40.543375 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:11:40.543406 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:11:40.545099 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:11:40.545620 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:11:40.545624 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:11:40.545626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:11:40.545721 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:11:40.545997 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:11:40.546001 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:11:40.546002 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:11:40.546259 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:11:40.546397 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:40.546643 (+ 246us) server_negotiation.cc:299] Negotiation successful
0620 19:11:40.546776 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207}
I20260620 19:11:40.550920 21052 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57966:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:11:40.555744 21052 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:11:40.573421 21548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:40.565587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.65:35935 (local address 127.0.0.1:57032)
0620 19:11:40.567690 (+ 2103us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:40.567701 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:11:40.567784 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:11:40.568498 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:11:40.568505 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:11:40.568531 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:11:40.568671 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:40.568679 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:40.570073 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:40.570077 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:40.571456 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:40.571469 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:40.572864 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:40.572868 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:40.572985 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:40.573022 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 19:11:40.573129 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 19:11:40.573265 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2047,"thread_start_us":94,"threads_started":1}
I20260620 19:11:40.574015 21550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:40.566328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57032 (local address 127.9.155.65:35935)
0620 19:11:40.567566 (+ 1238us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:40.567573 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:40.567866 (+ 293us) server_negotiation.cc:407] Connection header received
0620 19:11:40.567944 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:40.567949 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:40.568165 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:40.568322 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:11:40.568807 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:40.569942 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:40.571628 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:40.572537 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:40.572642 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:40.572801 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:40.573761 (+ 960us) server_negotiation.cc:299] Negotiation successful
0620 19:11:40.573878 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1139,"thread_start_us":104,"threads_started":1}
I20260620 19:11:40.577529 21197 tablet_service.cc:1467] Processing CreateTablet for tablet 4f3cc8ed74b4468799e018e4f0bf26d4 (DEFAULT_TABLE table=test-table [id=523f2a86f3e34627b9df825fd36a3119]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:11:40.578806 21197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f3cc8ed74b4468799e018e4f0bf26d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:11:40.580476 21552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:40.567044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.66:39907 (local address 127.0.0.1:52554)
0620 19:11:40.572695 (+ 5651us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:40.572709 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:11:40.572784 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:11:40.574015 (+ 1231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:11:40.574018 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:11:40.574034 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:11:40.574114 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:40.574131 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:40.576513 (+ 2382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:40.576517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:40.577783 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:40.577795 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:40.580024 (+ 2229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:40.580028 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:40.580132 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:40.580156 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 19:11:40.580257 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 19:11:40.580332 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5581,"thread_start_us":103,"threads_started":1}
I20260620 19:11:40.581001 21553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:40.567579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52554 (local address 127.9.155.66:39907)
0620 19:11:40.573040 (+ 5461us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:40.573051 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:40.573077 (+ 26us) server_negotiation.cc:407] Connection header received
0620 19:11:40.573166 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:40.573172 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:40.573399 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:40.573877 (+ 478us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:11:40.574257 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:40.576376 (+ 2119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:40.577950 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:40.579722 (+ 1772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:40.579823 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:40.579965 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:40.580752 (+ 787us) server_negotiation.cc:299] Negotiation successful
0620 19:11:40.580859 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5344,"thread_start_us":141,"threads_started":1}
I20260620 19:11:40.582759 21549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:40.565534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.67:37709 (local address 127.0.0.1:57228)
0620 19:11:40.567131 (+ 1597us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:40.567193 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 19:11:40.567309 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:11:40.572331 (+ 5022us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:11:40.572335 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:11:40.572357 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:11:40.572458 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:40.572465 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:40.580451 (+ 7986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:40.580454 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:40.581500 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:40.581510 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:40.582385 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:40.582389 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:40.582472 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:40.582493 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 19:11:40.582563 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 19:11:40.582634 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1426,"thread_start_us":98,"threads_started":1}
I20260620 19:11:40.582808 21551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:40.566349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57228 (local address 127.9.155.67:37709)
0620 19:11:40.570283 (+ 3934us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:40.570296 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:40.570323 (+ 27us) server_negotiation.cc:407] Connection header received
0620 19:11:40.570400 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:40.570406 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:40.570616 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:40.570790 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:11:40.579241 (+ 8451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:40.580333 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:40.581693 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:40.582265 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:40.582363 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:40.582509 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:40.582589 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 19:11:40.582685 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3818,"thread_start_us":141,"threads_started":1}
I20260620 19:11:40.584129 21336 tablet_service.cc:1467] Processing CreateTablet for tablet 4f3cc8ed74b4468799e018e4f0bf26d4 (DEFAULT_TABLE table=test-table [id=523f2a86f3e34627b9df825fd36a3119]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:11:40.585309 21336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f3cc8ed74b4468799e018e4f0bf26d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:11:40.585816 21470 tablet_service.cc:1467] Processing CreateTablet for tablet 4f3cc8ed74b4468799e018e4f0bf26d4 (DEFAULT_TABLE table=test-table [id=523f2a86f3e34627b9df825fd36a3119]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:11:40.586823 21470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f3cc8ed74b4468799e018e4f0bf26d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:11:40.596098 21556 tablet_bootstrap.cc:492] T 4f3cc8ed74b4468799e018e4f0bf26d4 P 5a43b5c5d38e4c85941380f50a90f1c6: Bootstrap starting.
I20260620 19:11:40.596575 21554 tablet_bootstrap.cc:492] T 4f3cc8ed74b4468799e018e4f0bf26d4 P 49652bd0070844ceba74158fb694a66d: Bootstrap starting.
I20260620 19:11:40.599004 21554 tablet_bootstrap.cc:654] T 4f3cc8ed74b4468799e018e4f0bf26d4 P 49652bd0070844ceba74158fb694a66d: Neither blocks nor log segments found. Creating new log.
I20260620 19:11:40.599947 21554 log.cc:824] T 4f3cc8ed74b4468799e018e4f0bf26d4 P 49652bd0070844ceba74158fb694a66d: Log is configured to *not* fsync() on all Append() calls
I20260620 19:11:40.602162 21556 tablet_bootstrap.cc:654] T 4f3cc8ed74b4468799e018e4f0bf26d4 P 5a43b5c5d38e4c85941380f50a90f1c6: Neither blocks nor log segments found. Creating new log.
I20260620 19:11:40.602456 21554 tablet_bootstrap.cc:492] T 4f3cc8ed74b4468799e018e4f0bf26d4 P 49652bd0070844ceba74158fb694a66d: No bootstrap required, opened a new log
I20260620 19:11:40.602649 21554 ts_tablet_manager.cc:1397] T 4f3cc8ed74b4468799e018e4f0bf26d4 P 49652bd0070844ceba74158fb694a66d: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 19:11:40.603041 21556 log.cc:824] T 4f3cc8ed74b4468799e018e4f0bf26d4 P 5a43b5c5d38e4c85941380f50a90f1c6: Log is configured to *not* fsync() on all Append() calls
I20260620 19:11:40.603822 21555 tablet_bootstrap.cc:492] T 4f3cc8ed74b4468799e018e4f0bf26d4 P 22fbb8de4a52429fbc4f887f161322d4: Bootstrap starting.
I20260620 19:11:40.606326 21555 tablet_bootstrap.cc:654] T 4f3cc8ed74b4468799e018e4f0bf26d4 P 22fbb8de4a52429fbc4f887f161322d4: Neither blocks nor log segments found. Creating new log.
I20260620 19:11:40.607760 21555 log.cc:824] T 4f3cc8ed74b4468799e018e4f0bf26d4 P 22fbb8de4a52429fbc4f887f161322d4: Log is configured to *not* fsync() on all Append() calls
I20260620 19:11:40.607688 21554 raft_consensus.cc:348] T 4f3cc8ed74b4468799e018e4f0bf26d4 P 49652bd0070844ceba74158fb694a66d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a43b5c5d38e4c85941380f50a90f1c6" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 37709 } } peers { permanent_uuid: "49652bd0070844ceba74158fb694a66d" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 35935 } } peers { permanent_uuid: "22fbb8de4a52429fbc4f887f161322d4" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 39907 } }
I20260620 19:11:40.608126 21554 raft_consensus.cc:374] T 4f3cc8ed74b4468799e018e4f0bf26d4 P 49652bd0070844ceba74158fb694a66d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:11:40.608300 21554 raft_consensus.cc:729] T 4f3cc8ed74b4468799e018e4f0bf26d4 P 49652bd0070844ceba74158fb694a66d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49652bd0070844ceba74158fb694a66d, State: Initialized, Role: FOLLOWER
I20260620 19:11:40.609012 21554 consensus_queue.cc:260] T 4f3cc8ed74b4468799e018e4f0bf26d4 P 49652bd0070844ceba74158fb694a66d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a43b5c5d38e4c85941380f50a90f1c6" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 37709 } } peers { permanent_uuid: "49652bd0070844ceba74158fb694a66d" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 35935 } } peers { permanent_uuid: "22fbb8de4a52429fbc4f887f161322d4" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 39907 } }
I20260620 19:11:40.610203 21554 ts_tablet_manager.cc:1428] T 4f3cc8ed74b4468799e018e4f0bf26d4 P 49652bd0070844ceba74158fb694a66d: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260620 19:11:40.610359 21556 tablet_bootstrap.cc:492] T 4f3cc8ed74b4468799e018e4f0bf26d4 P 5a43b5c5d38e4c85941380f50a90f1c6: No bootstrap required, opened a new log
I20260620 19:11:40.610572 21556 ts_tablet_manager.cc:1397] T 4f3cc8ed74b4468799e018e4f0bf26d4 P 5a43b5c5d38e4c85941380f50a90f1c6: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.000s
I20260620 19:11:40.610585 21266 heartbeater.cc:503] Master 127.9.155.126:42645 was elected leader, sending a full tablet report...
I20260620 19:11:40.615495 21556 raft_consensus.cc:348] T 4f3cc8ed74b4468799e018e4f0bf26d4 P 5a43b5c5d38e4c85941380f50a90f1c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a43b5c5d38e4c85941380f50a90f1c6" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 37709 } } peers { permanent_uuid: "49652bd0070844ceba74158fb694a66d" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 35935 } } peers { permanent_uuid: "22fbb8de4a52429fbc4f887f161322d4" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 39907 } }
I20260620 19:11:40.615913 21556 raft_consensus.cc:374] T 4f3cc8ed74b4468799e018e4f0bf26d4 P 5a43b5c5d38e4c85941380f50a90f1c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:11:40.616012 21556 raft_consensus.cc:729] T 4f3cc8ed74b4468799e018e4f0bf26d4 P 5a43b5c5d38e4c85941380f50a90f1c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a43b5c5d38e4c85941380f50a90f1c6, State: Initialized, Role: FOLLOWER
I20260620 19:11:40.616611 21556 consensus_queue.cc:260] T 4f3cc8ed74b4468799e018e4f0bf26d4 P 5a43b5c5d38e4c85941380f50a90f1c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a43b5c5d38e4c85941380f50a90f1c6" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 37709 } } peers { permanent_uuid: "49652bd0070844ceba74158fb694a66d" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 35935 } } peers { permanent_uuid: "22fbb8de4a52429fbc4f887f161322d4" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 39907 } }
I20260620 19:11:40.617352 21536 heartbeater.cc:503] Master 127.9.155.126:42645 was elected leader, sending a full tablet report...
I20260620 19:11:40.617870 21556 ts_tablet_manager.cc:1428] T 4f3cc8ed74b4468799e018e4f0bf26d4 P 5a43b5c5d38e4c85941380f50a90f1c6: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260620 19:11:40.620057 21555 tablet_bootstrap.cc:492] T 4f3cc8ed74b4468799e018e4f0bf26d4 P 22fbb8de4a52429fbc4f887f161322d4: No bootstrap required, opened a new log
I20260620 19:11:40.620302 21555 ts_tablet_manager.cc:1397] T 4f3cc8ed74b4468799e018e4f0bf26d4 P 22fbb8de4a52429fbc4f887f161322d4: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.002s
I20260620 19:11:40.624413 21555 raft_consensus.cc:348] T 4f3cc8ed74b4468799e018e4f0bf26d4 P 22fbb8de4a52429fbc4f887f161322d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a43b5c5d38e4c85941380f50a90f1c6" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 37709 } } peers { permanent_uuid: "49652bd0070844ceba74158fb694a66d" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 35935 } } peers { permanent_uuid: "22fbb8de4a52429fbc4f887f161322d4" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 39907 } }
I20260620 19:11:40.624735 21555 raft_consensus.cc:374] T 4f3cc8ed74b4468799e018e4f0bf26d4 P 22fbb8de4a52429fbc4f887f161322d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:11:40.624835 21555 raft_consensus.cc:729] T 4f3cc8ed74b4468799e018e4f0bf26d4 P 22fbb8de4a52429fbc4f887f161322d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22fbb8de4a52429fbc4f887f161322d4, State: Initialized, Role: FOLLOWER
I20260620 19:11:40.625349 21555 consensus_queue.cc:260] T 4f3cc8ed74b4468799e018e4f0bf26d4 P 22fbb8de4a52429fbc4f887f161322d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a43b5c5d38e4c85941380f50a90f1c6" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 37709 } } peers { permanent_uuid: "49652bd0070844ceba74158fb694a66d" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 35935 } } peers { permanent_uuid: "22fbb8de4a52429fbc4f887f161322d4" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 39907 } }
I20260620 19:11:40.626161 21402 heartbeater.cc:503] Master 127.9.155.126:42645 was elected leader, sending a full tablet report...
I20260620 19:11:40.626583 21555 ts_tablet_manager.cc:1428] T 4f3cc8ed74b4468799e018e4f0bf26d4 P 22fbb8de4a52429fbc4f887f161322d4: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
Jun 20 19:11:40 dist-test-slave-n3fx krb5kdc[20576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982699, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.126@KRBTEST.COM for HTTP/127.9.155.84@KRBTEST.COM
W20260620 19:11:40.662933 21403 tablet.cc:2369] T 4f3cc8ed74b4468799e018e4f0bf26d4 P 22fbb8de4a52429fbc4f887f161322d4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:11:40.737299 21560 raft_consensus.cc:482] T 4f3cc8ed74b4468799e018e4f0bf26d4 P 49652bd0070844ceba74158fb694a66d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:11:40.737555 21560 raft_consensus.cc:504] T 4f3cc8ed74b4468799e018e4f0bf26d4 P 49652bd0070844ceba74158fb694a66d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a43b5c5d38e4c85941380f50a90f1c6" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 37709 } } peers { permanent_uuid: "49652bd0070844ceba74158fb694a66d" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 35935 } } peers { permanent_uuid: "22fbb8de4a52429fbc4f887f161322d4" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 39907 } }
I20260620 19:11:40.738976 21560 leader_election.cc:288] T 4f3cc8ed74b4468799e018e4f0bf26d4 P 49652bd0070844ceba74158fb694a66d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a43b5c5d38e4c85941380f50a90f1c6 (127.9.155.67:37709), 22fbb8de4a52429fbc4f887f161322d4 (127.9.155.66:39907)
I20260620 19:11:40.744963 21553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:40.740378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.65:40783 (local address 127.9.155.66:39907)
0620 19:11:40.740488 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:40.740492 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:40.740754 (+ 262us) server_negotiation.cc:407] Connection header received
0620 19:11:40.740928 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:40.740931 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:40.740977 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:40.741021 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:11:40.741439 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:40.742192 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:40.743708 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:40.744297 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:40.744389 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:40.744442 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:40.744753 (+ 311us) server_negotiation.cc:299] Negotiation successful
0620 19:11:40.744832 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 19:11:40.745597 21566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:40.739776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.66:39907 (local address 127.9.155.65:40783)
0620 19:11:40.740673 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:40.740686 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:11:40.740819 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:11:40.741158 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:11:40.741161 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:11:40.741188 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:11:40.741324 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:40.741330 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:40.742327 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:40.742330 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:40.743570 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:40.743581 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:40.744536 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:40.744539 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:40.744631 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:40.744652 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 19:11:40.745372 (+ 720us) client_negotiation.cc:240] Negotiation successful
0620 19:11:40.745444 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":824,"thread_start_us":78,"threads_started":1}
I20260620 19:11:40.746356 21551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:40.740255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.65:47225 (local address 127.9.155.67:37709)
0620 19:11:40.740383 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:40.740388 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:40.740404 (+ 16us) server_negotiation.cc:407] Connection header received
0620 19:11:40.740466 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:40.740469 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:40.740519 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:40.740612 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:11:40.742360 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:40.743093 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:40.745091 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:40.745651 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:40.745750 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:40.745798 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:40.746151 (+ 353us) server_negotiation.cc:299] Negotiation successful
0620 19:11:40.746233 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 19:11:40.746308 21352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f3cc8ed74b4468799e018e4f0bf26d4" candidate_uuid: "49652bd0070844ceba74158fb694a66d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fbb8de4a52429fbc4f887f161322d4" is_pre_election: true
I20260620 19:11:40.746661 21352 raft_consensus.cc:2454] T 4f3cc8ed74b4468799e018e4f0bf26d4 P 22fbb8de4a52429fbc4f887f161322d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49652bd0070844ceba74158fb694a66d in term 0.
I20260620 19:11:40.746802 21565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:40.739382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.67:37709 (local address 127.9.155.65:47225)
0620 19:11:40.739721 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:40.739738 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 19:11:40.739864 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:11:40.740940 (+ 1076us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:11:40.740943 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:11:40.740965 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:11:40.741544 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:40.741552 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:40.743685 (+ 2133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:40.743688 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:40.744929 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:11:40.744939 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:11:40.745892 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:11:40.745896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:11:40.745992 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:40.746013 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 19:11:40.746588 (+ 575us) client_negotiation.cc:240] Negotiation successful
0620 19:11:40.746661 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":128,"threads_started":1}
I20260620 19:11:40.747154 21156 leader_election.cc:302] T 4f3cc8ed74b4468799e018e4f0bf26d4 P 49652bd0070844ceba74158fb694a66d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fbb8de4a52429fbc4f887f161322d4, 49652bd0070844ceba74158fb694a66d; no voters:
I20260620 19:11:40.747447 21560 raft_consensus.cc:2781] T 4f3cc8ed74b4468799e018e4f0bf26d4 P 49652bd0070844ceba74158fb694a66d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:11:40.747540 21560 raft_consensus.cc:482] T 4f3cc8ed74b4468799e018e4f0bf26d4 P 49652bd0070844ceba74158fb694a66d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:11:40.747579 21560 raft_consensus.cc:3037] T 4f3cc8ed74b4468799e018e4f0bf26d4 P 49652bd0070844ceba74158fb694a66d [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:11:40.748513 21490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f3cc8ed74b4468799e018e4f0bf26d4" candidate_uuid: "49652bd0070844ceba74158fb694a66d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a43b5c5d38e4c85941380f50a90f1c6" is_pre_election: true
I20260620 19:11:40.748798 21490 raft_consensus.cc:2454] T 4f3cc8ed74b4468799e018e4f0bf26d4 P 5a43b5c5d38e4c85941380f50a90f1c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49652bd0070844ceba74158fb694a66d in term 0.
I20260620 19:11:40.750245 21560 raft_consensus.cc:504] T 4f3cc8ed74b4468799e018e4f0bf26d4 P 49652bd0070844ceba74158fb694a66d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a43b5c5d38e4c85941380f50a90f1c6" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 37709 } } peers { permanent_uuid: "49652bd0070844ceba74158fb694a66d" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 35935 } } peers { permanent_uuid: "22fbb8de4a52429fbc4f887f161322d4" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 39907 } }
I20260620 19:11:40.750609 21560 leader_election.cc:288] T 4f3cc8ed74b4468799e018e4f0bf26d4 P 49652bd0070844ceba74158fb694a66d [CANDIDATE]: Term 1 election: Requested vote from peers 5a43b5c5d38e4c85941380f50a90f1c6 (127.9.155.67:37709), 22fbb8de4a52429fbc4f887f161322d4 (127.9.155.66:39907)
I20260620 19:11:40.751078 21352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f3cc8ed74b4468799e018e4f0bf26d4" candidate_uuid: "49652bd0070844ceba74158fb694a66d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fbb8de4a52429fbc4f887f161322d4"
I20260620 19:11:40.751199 21352 raft_consensus.cc:3037] T 4f3cc8ed74b4468799e018e4f0bf26d4 P 22fbb8de4a52429fbc4f887f161322d4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:11:40.752437 21352 raft_consensus.cc:2454] T 4f3cc8ed74b4468799e018e4f0bf26d4 P 22fbb8de4a52429fbc4f887f161322d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49652bd0070844ceba74158fb694a66d in term 1.
I20260620 19:11:40.753017 21156 leader_election.cc:302] T 4f3cc8ed74b4468799e018e4f0bf26d4 P 49652bd0070844ceba74158fb694a66d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22fbb8de4a52429fbc4f887f161322d4, 49652bd0070844ceba74158fb694a66d; no voters:
I20260620 19:11:40.753310 21490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f3cc8ed74b4468799e018e4f0bf26d4" candidate_uuid: "49652bd0070844ceba74158fb694a66d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a43b5c5d38e4c85941380f50a90f1c6"
I20260620 19:11:40.753436 21490 raft_consensus.cc:3037] T 4f3cc8ed74b4468799e018e4f0bf26d4 P 5a43b5c5d38e4c85941380f50a90f1c6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:11:40.754694 21490 raft_consensus.cc:2454] T 4f3cc8ed74b4468799e018e4f0bf26d4 P 5a43b5c5d38e4c85941380f50a90f1c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49652bd0070844ceba74158fb694a66d in term 1.
I20260620 19:11:40.755205 21560 raft_consensus.cc:2781] T 4f3cc8ed74b4468799e018e4f0bf26d4 P 49652bd0070844ceba74158fb694a66d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:11:40.755429 21560 raft_consensus.cc:686] T 4f3cc8ed74b4468799e018e4f0bf26d4 P 49652bd0070844ceba74158fb694a66d [term 1 LEADER]: Becoming Leader. State: Replica: 49652bd0070844ceba74158fb694a66d, State: Running, Role: LEADER
I20260620 19:11:40.755780 21560 consensus_queue.cc:237] T 4f3cc8ed74b4468799e018e4f0bf26d4 P 49652bd0070844ceba74158fb694a66d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a43b5c5d38e4c85941380f50a90f1c6" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 37709 } } peers { permanent_uuid: "49652bd0070844ceba74158fb694a66d" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 35935 } } peers { permanent_uuid: "22fbb8de4a52429fbc4f887f161322d4" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 39907 } }
W20260620 19:11:40.757737 21537 tablet.cc:2369] T 4f3cc8ed74b4468799e018e4f0bf26d4 P 5a43b5c5d38e4c85941380f50a90f1c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:11:40.759490 21052 catalog_manager.cc:5495] T 4f3cc8ed74b4468799e018e4f0bf26d4 P 49652bd0070844ceba74158fb694a66d reported cstate change: term changed from 0 to 1, leader changed from <none> to 49652bd0070844ceba74158fb694a66d (127.9.155.65). New cstate: current_term: 1 leader_uuid: "49652bd0070844ceba74158fb694a66d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a43b5c5d38e4c85941380f50a90f1c6" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 37709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49652bd0070844ceba74158fb694a66d" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 35935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22fbb8de4a52429fbc4f887f161322d4" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 39907 } health_report { overall_health: UNKNOWN } } }
I20260620 19:11:40.772856 21550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:40.767845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57038 (local address 127.9.155.65:35935)
0620 19:11:40.768247 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0620 19:11:40.768252 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:11:40.768266 (+ 14us) server_negotiation.cc:407] Connection header received
0620 19:11:40.768340 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:11:40.768342 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:11:40.768405 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:11:40.768509 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:11:40.768811 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:40.769544 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:40.770792 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:11:40.771135 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:11:40.771226 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:11:40.771627 (+ 401us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:11:40.772188 (+ 561us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:11:40.772335 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 19:11:40.772656 (+ 321us) server_negotiation.cc:299] Negotiation successful
0620 19:11:40.772717 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294}
I20260620 19:11:40.780974 21490 raft_consensus.cc:1261] T 4f3cc8ed74b4468799e018e4f0bf26d4 P 5a43b5c5d38e4c85941380f50a90f1c6 [term 1 FOLLOWER]: Refusing update from remote peer 49652bd0070844ceba74158fb694a66d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:11:40.781901 21352 raft_consensus.cc:1261] T 4f3cc8ed74b4468799e018e4f0bf26d4 P 22fbb8de4a52429fbc4f887f161322d4 [term 1 FOLLOWER]: Refusing update from remote peer 49652bd0070844ceba74158fb694a66d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:11:40.782873 21567 consensus_queue.cc:1035] T 4f3cc8ed74b4468799e018e4f0bf26d4 P 49652bd0070844ceba74158fb694a66d [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fbb8de4a52429fbc4f887f161322d4" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 39907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:11:40.783071 21560 consensus_queue.cc:1035] T 4f3cc8ed74b4468799e018e4f0bf26d4 P 49652bd0070844ceba74158fb694a66d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a43b5c5d38e4c85941380f50a90f1c6" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 37709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:11:40.798794 21570 mvcc.cc:205] Tried to move back new op lower bound from 7299001142388514816 to 7299001142299885568. Current Snapshot: MvccSnapshot[applied={T|T < 7299001142388514816}]
I20260620 19:11:40.803962 21573 mvcc.cc:205] Tried to move back new op lower bound from 7299001142388514816 to 7299001142299885568. Current Snapshot: MvccSnapshot[applied={T|T < 7299001142388514816}]
I20260620 19:11:40.806200 21571 mvcc.cc:205] Tried to move back new op lower bound from 7299001142388514816 to 7299001142299885568. Current Snapshot: MvccSnapshot[applied={T|T < 7299001142388514816}]
I20260620 19:11:40.816094 21052 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57966:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:11:40.816313 21052 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57966:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:11:40.819207 21052 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e2b28111327940e3a1579d1fe9ba6e3e: Sending DeleteTablet for 3 replicas of tablet 4f3cc8ed74b4468799e018e4f0bf26d4
I20260620 19:11:40.820541 21336 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f3cc8ed74b4468799e018e4f0bf26d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:11:40 UTC) from {username='kudu', principal='kudu/127.9.155.126@KRBTEST.COM'} at 127.0.0.1:52554
I20260620 19:11:40.821067 21197 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f3cc8ed74b4468799e018e4f0bf26d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:11:40 UTC) from {username='kudu', principal='kudu/127.9.155.126@KRBTEST.COM'} at 127.0.0.1:57032
I20260620 19:11:40.821233 21470 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f3cc8ed74b4468799e018e4f0bf26d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:11:40 UTC) from {username='kudu', principal='kudu/127.9.155.126@KRBTEST.COM'} at 127.0.0.1:57228
I20260620 19:11:40.822000 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 21133
I20260620 19:11:40.826505 21581 tablet_replica.cc:330] T 4f3cc8ed74b4468799e018e4f0bf26d4 P 5a43b5c5d38e4c85941380f50a90f1c6: stopping tablet replica
I20260620 19:11:40.826969 21581 raft_consensus.cc:2229] T 4f3cc8ed74b4468799e018e4f0bf26d4 P 5a43b5c5d38e4c85941380f50a90f1c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:11:40.827426 21581 raft_consensus.cc:2258] T 4f3cc8ed74b4468799e018e4f0bf26d4 P 5a43b5c5d38e4c85941380f50a90f1c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:11:40.828002 21579 tablet_replica.cc:330] T 4f3cc8ed74b4468799e018e4f0bf26d4 P 22fbb8de4a52429fbc4f887f161322d4: stopping tablet replica
I20260620 19:11:40.828429 21579 raft_consensus.cc:2229] T 4f3cc8ed74b4468799e018e4f0bf26d4 P 22fbb8de4a52429fbc4f887f161322d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:11:40.828774 21579 raft_consensus.cc:2258] T 4f3cc8ed74b4468799e018e4f0bf26d4 P 22fbb8de4a52429fbc4f887f161322d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:11:40.829105 21581 ts_tablet_manager.cc:1905] T 4f3cc8ed74b4468799e018e4f0bf26d4 P 5a43b5c5d38e4c85941380f50a90f1c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:11:40.831780 21581 ts_tablet_manager.cc:1918] T 4f3cc8ed74b4468799e018e4f0bf26d4 P 5a43b5c5d38e4c85941380f50a90f1c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:11:40.831914 21581 log.cc:1196] T 4f3cc8ed74b4468799e018e4f0bf26d4 P 5a43b5c5d38e4c85941380f50a90f1c6: Deleting WAL directory at /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982595231230-9837-0/minicluster-data/ts-2/wal/wals/4f3cc8ed74b4468799e018e4f0bf26d4
I20260620 19:11:40.833016 21581 ts_tablet_manager.cc:1939] T 4f3cc8ed74b4468799e018e4f0bf26d4 P 5a43b5c5d38e4c85941380f50a90f1c6: Deleting consensus metadata
I20260620 19:11:40.834388 21039 catalog_manager.cc:4841] TS 5a43b5c5d38e4c85941380f50a90f1c6 (127.9.155.67:37709): tablet 4f3cc8ed74b4468799e018e4f0bf26d4 (table test-table [id=523f2a86f3e34627b9df825fd36a3119]) successfully deleted
W20260620 19:11:40.836601 21037 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.155.65:35935 (error 108)
I20260620 19:11:40.837877 21549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:40.837595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.65:35935 (local address 127.0.0.1:57040)
0620 19:11:40.837725 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:40.837792 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.65:35935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 19:11:40.838017 21037 catalog_manager.cc:4584] TS 49652bd0070844ceba74158fb694a66d (127.9.155.65:35935): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4f3cc8ed74b4468799e018e4f0bf26d4: Network error: Client connection negotiation failed: client connection to 127.9.155.65:35935: connect: Connection refused (error 111)
I20260620 19:11:40.840504 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 21272
I20260620 19:11:40.846691 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 21407
I20260620 19:11:40.846972 21549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:11:40.846703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.66:39907 (local address 127.0.0.1:52560)
0620 19:11:40.846831 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 19:11:40.846887 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.66:39907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 19:11:40.855140 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 21020
I20260620 19:11:40.862401 9837 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 19:11:41.197693 9837 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 19:11:41.197815 9837 mini_ranger.cc:66] Stopping Ranger...
I20260620 19:11:41.533051 9837 mini_ranger.cc:68] Stopped Ranger
2026-06-20 19:11:41.533 UTC [20606] LOG: received smart shutdown request
2026-06-20 19:11:41.536 UTC [20606] LOG: background worker "logical replication launcher" (PID 20615) exited with exit code 1
2026-06-20 19:11:41.536 UTC [20610] LOG: shutting down
2026-06-20 19:11:41.693 UTC [20606] LOG: database system is shut down
2026-06-20T19:11:41Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46669 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:11:41 dist-test-slave-n3fx krb5kdc[21591](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:11:41 dist-test-slave-n3fx krb5kdc[21591](info): set up 2 sockets
Jun 20 19:11:41 dist-test-slave-n3fx krb5kdc[21591](info): commencing operation
krb5kdc: starting...
W20260620 19:11:42.740144 9837 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/krb5kdc/test-user.keytab.
Jun 20 19:11:42 dist-test-slave-n3fx krb5kdc[21591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:11:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:11:42Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.9.155.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.9.155.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.9.155.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/krb5kdc/rangeradmin_127.9.155.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.9.155.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/krb5kdc/rangeradmin_127.9.155.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.9.155.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.9.155.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.9.155.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/krb5kdc/rangerlookup_127.9.155.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.9.155.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/krb5kdc/rangerlookup_127.9.155.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.9.155.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.155.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.155.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/krb5kdc/HTTP_127.9.155.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.9.155.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/krb5kdc/HTTP_127.9.155.84@KRBTEST.COM.keytab.
I20260620 19:11:42.902365 9837 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task74hYKn/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/postgres -l logfile start
2026-06-20 19:11:45.328 UTC [21621] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 19:11:45.328 UTC [21621] LOG: listening on IPv4 address "127.9.155.84", port 48467
2026-06-20 19:11:45.332 UTC [21621] LOG: listening on Unix socket "/tmp/.s.PGSQL.48467"
2026-06-20 19:11:45.352 UTC [21624] LOG: database system was shut down at 2026-06-20 19:11:43 UTC
2026-06-20 19:11:45.355 UTC [21621] LOG: database system is ready to accept connections
I20260620 19:11:46.231907 9837 mini_postgres.cc:96] Postgres bound to 48467
2026-06-20 19:11:46.237 UTC [21632] FATAL: database "slave" does not exist
127.9.155.84:48467 - accepting connections
I20260620 19:11:46.238256 9837 mini_ranger.cc:161] Starting Ranger...
I20260620 19:11:46.253291 9837 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 19:11:46.720559 9837 mini_ranger.cc:87] Created ranger Postgres database
I20260620 19:11:46.720635 9837 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ranger-admin
2026-06-20 19:11:46,986 [I] DB FLAVOR :POSTGRES
2026-06-20 19:11:46,987 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 19:11:46,987 [I] Checking connection..
2026-06-20 19:11:46,987 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task74hYKn/build/debug/bin/postgresql.jar:/tmp/dist-test-task74hYKn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.155.84:48467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 19:11:47,247 [I] Checking connection passed.
2026-06-20 19:11:47,248 [I] --------- Verifying version history table ---------
2026-06-20 19:11:47,248 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task74hYKn/build/debug/bin/postgresql.jar:/tmp/dist-test-task74hYKn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.155.84:48467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 19:11:47,493 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 19:11:47,493 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task74hYKn/build/debug/bin/postgresql.jar:/tmp/dist-test-task74hYKn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.155.84:48467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 19:11:47,733 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 19:11:47,733 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 19:11:47,733 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task74hYKn/build/debug/bin/postgresql.jar:/tmp/dist-test-task74hYKn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.155.84:48467/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task74hYKn/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 19:11:47.960 UTC [21696] WARNING: there is no transaction in progress
2026-06-20 19:11:47,966 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 19:11:47,966 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task74hYKn/build/debug/bin/postgresql.jar:/tmp/dist-test-task74hYKn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.155.84:48467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 19:11:48,201 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 19:11:48,201 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 19:11:48,202 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task74hYKn/build/debug/bin/postgresql.jar:/tmp/dist-test-task74hYKn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.155.84:48467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 19:11:48,437 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task74hYKn/build/debug/bin/postgresql.jar:/tmp/dist-test-task74hYKn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.155.84:48467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 19:11:48,678 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task74hYKn/build/debug/bin/postgresql.jar:/tmp/dist-test-task74hYKn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.155.84:48467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-n3fx.c.gcp-upstream.internal','N') ;"
2026-06-20 19:11:48,911 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task74hYKn/build/debug/bin/postgresql.jar:/tmp/dist-test-task74hYKn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.155.84:48467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 19:11:49,175 [I] Table x_portal_user does not exist in database ranger
2026-06-20 19:11:49,176 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task74hYKn/build/debug/bin/postgresql.jar:/tmp/dist-test-task74hYKn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.155.84:48467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 19:11:49,410 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 19:11:49,410 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task74hYKn/build/debug/bin/postgresql.jar:/tmp/dist-test-task74hYKn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.155.84:48467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 19:11:49,651 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task74hYKn/build/debug/bin/postgresql.jar:/tmp/dist-test-task74hYKn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.155.84:48467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 19:11:49,902 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 19:11:49,902 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task74hYKn/build/debug/bin/postgresql.jar:/tmp/dist-test-task74hYKn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.155.84:48467/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task74hYKn/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 19:11:50.891 UTC [21804] WARNING: there is no transaction in progress
2026-06-20 19:11:50.908 UTC [21804] WARNING: there is no transaction in progress
2026-06-20 19:11:50.926 UTC [21804] WARNING: there is no transaction in progress
2026-06-20 19:11:50.941 UTC [21804] WARNING: there is no transaction in progress
2026-06-20 19:11:50.959 UTC [21804] WARNING: there is no transaction in progress
2026-06-20 19:11:50.981 UTC [21804] WARNING: there is no transaction in progress
2026-06-20 19:11:51.051 UTC [21804] WARNING: there is no transaction in progress
2026-06-20 19:11:51.058 UTC [21804] WARNING: there is no transaction in progress
2026-06-20 19:11:51.074 UTC [21804] WARNING: there is no transaction in progress
2026-06-20 19:11:51.085 UTC [21804] WARNING: there is no transaction in progress
2026-06-20 19:11:51.095 UTC [21804] WARNING: there is no transaction in progress
2026-06-20 19:11:51.111 UTC [21804] WARNING: there is no transaction in progress
2026-06-20 19:11:51.121 UTC [21804] WARNING: there is no transaction in progress
2026-06-20 19:11:51.127 UTC [21804] WARNING: there is no transaction in progress
2026-06-20 19:11:51.134 UTC [21804] WARNING: there is no transaction in progress
2026-06-20 19:11:51.141 UTC [21804] WARNING: there is no transaction in progress
2026-06-20 19:11:51,787 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 19:11:51,788 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task74hYKn/build/debug/bin/postgresql.jar:/tmp/dist-test-task74hYKn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.155.84:48467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 19:11:52,009 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 19:11:52,010 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task74hYKn/build/debug/bin/postgresql.jar:/tmp/dist-test-task74hYKn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.155.84:48467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 19:11:52,272 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 19:11:52,272 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task74hYKn/build/debug/bin/postgresql.jar:/tmp/dist-test-task74hYKn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.155.84:48467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 19:11:52,519 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 19:11:52,519 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task74hYKn/build/debug/bin/postgresql.jar:/tmp/dist-test-task74hYKn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.155.84:48467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 19:11:52,778 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task74hYKn/build/debug/bin/postgresql.jar:/tmp/dist-test-task74hYKn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.155.84:48467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 19:11:53,030 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task74hYKn/build/debug/bin/postgresql.jar:/tmp/dist-test-task74hYKn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.155.84:48467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-n3fx.c.gcp-upstream.internal';"
2026-06-20 19:11:53,264 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 19:11:53,265 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task74hYKn/build/debug/bin/postgresql.jar:/tmp/dist-test-task74hYKn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.155.84:48467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 19:11:53,519 [I] DB_PATCHES have already been applied
I20260620 19:11:53.525426 9837 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ranger-admin:/tmp/dist-test-task74hYKn/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task74hYKn/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task74hYKn/build/debug/bin/postgresql.jar:/tmp/dist-test-task74hYKn/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 19:11:53.525492 9837 mini_ranger.cc:193] Using host: 127.9.155.84
I20260620 19:11:53.528690 9837 mini_ranger.cc:227] Ranger admin URL: http://127.9.155.84:37829
Jun 20, 2026 7:11:53 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 7:11:53 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 7:11:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 7:11:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 7:11:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 7:11:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 7:11:54 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 19:11:54.545228 9837 mini_ranger.cc:160] Time spent starting Ranger: real 8.307s user 0.004s sys 0.005s
Jun 20, 2026 7:12:14 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 7:12:14 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.MetricsREST
Jun 20, 2026 7:12:14 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 7:12:14 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 7:12:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 7:12:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 7:12:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 7:12:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 7:12:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 7:12:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 7:12:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 7:12:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 7:12:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 7:12:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 7:12:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 7:12:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 7:12:14 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 7:12:15 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 19:12:15.781989 9837 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.9.155.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.9.155.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.9.155.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/krb5kdc/rangerkms_127.9.155.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.9.155.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/krb5kdc/rangerkms_127.9.155.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.9.155.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.9.155.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.155.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/krb5kdc/HTTP_127.9.155.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.9.155.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/krb5kdc/HTTP_127.9.155.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 19:12:15 dist-test-slave-n3fx krb5kdc[21591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982735, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 19:12:15.860775 9837 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 19:12:15.870091 9837 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 19:12:16.701726 9837 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 19:12:16.701794 9837 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ranger-kms
I20260620 19:12:16.701812 9837 mini_ranger_kms.cc:227] Using postgres at 127.9.155.84:48467
2026-06-20 19:12:16,864 [I] DB FLAVOR :POSTGRES
2026-06-20 19:12:16,865 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 19:12:16,865 [I] Checking connection
2026-06-20 19:12:16,865 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task74hYKn/build/debug/bin/postgresql.jar:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.155.84:48467/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 19:12:17,123 [I] connection success
2026-06-20 19:12:17,123 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 19:12:17,124 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 19:12:17,124 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task74hYKn/build/debug/bin/postgresql.jar:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.155.84:48467/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 19:12:17,366 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 19:12:17,366 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 19:12:17,366 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 19:12:17,367 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task74hYKn/build/debug/bin/postgresql.jar:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.155.84:48467/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 19:12:17,624 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 19:12:18.087733 9837 mini_ranger_kms.cc:310] Created kms service
I20260620 19:12:18.298228 9837 mini_ranger_kms.cc:326] Created kudu user
I20260620 19:12:18.340914 9837 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 19:12:18 dist-test-slave-n3fx krb5kdc[21591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982735, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.9.155.84@KRBTEST.COM
I20260620 19:12:18.754432 9837 mini_ranger_kms.cc:384] Added ranger policy
I20260620 19:12:18.754588 9837 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ranger-kms:/tmp/dist-test-task74hYKn/build/debug/bin/postgresql.jar:/tmp/dist-test-task74hYKn/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task74hYKn/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 19:12:18.754612 9837 mini_ranger_kms.cc:240] Using host: 127.9.155.84
I20260620 19:12:18.758261 9837 mini_ranger_kms.cc:276] Ranger KMS PID: 21984
I20260620 19:12:18.758353 9837 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.9.155.84:58483
Jun 20, 2026 7:12:19 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 7:12:19 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 7:12:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 7:12:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 7:12:20 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 7:12:20 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 19:12:20.775305 9837 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.915s user 0.011s sys 0.137s
I20260620 19:12:20.775434 9837 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 19:12:20.775508 9837 mini_ranger_kms.cc:401] 127.9.155.84:58483/kms/v1/keys
Jun 20 19:12:20 dist-test-slave-n3fx krb5kdc[21591](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781982740, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.155.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 19:12:21 dist-test-slave-n3fx krb5kdc[21591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982740, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.155.84@KRBTEST.COM for HTTP/127.9.155.84@KRBTEST.COM
Jun 20 19:12:21 dist-test-slave-n3fx krb5kdc[21591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982740, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.155.84@KRBTEST.COM for HTTP/127.9.155.84@KRBTEST.COM
Jun 20 19:12:23 dist-test-slave-n3fx krb5kdc[21591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.9.155.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.155.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.155.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab.
Entry for principal kudu/127.9.155.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab.
I20260620 19:12:23.986756 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:41439
--webserver_interface=127.9.155.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab
--principal=kudu/127.9.155.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:41329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.155.126:41439
--ranger_config_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.155.84:58483/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:12:24.102732 22035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:24.103019 22035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:24.103078 22035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:24.107416 22035 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:12:24.107481 22035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:24.107499 22035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:24.107515 22035 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:12:24.107529 22035 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:12:24.113346 22035 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:41329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/master-0/wal
--ranger_kms_url=127.9.155.84:58483/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.9.155.126:41439
--ranger_config_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab
--principal=kudu/127.9.155.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:41439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.155.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22035
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:12:24.114527 22035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:24.115586 22035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:12:24.124435 22041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:24.124411 22040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:24.124560 22043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:24.124818 22035 server_base.cc:1034] running on GCE node
I20260620 19:12:24.125408 22035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:24.126519 22035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:24.127712 22035 hybrid_clock.cc:648] HybridClock initialized: now 1781982744127692 us; error 37 us; skew 500 ppm
Jun 20 19:12:24 dist-test-slave-n3fx krb5kdc[21591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982744, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:24.130518 22035 init.cc:377] Logged in from keytab as kudu/127.9.155.126@KRBTEST.COM (short username kudu)
I20260620 19:12:24.131460 22035 webserver.cc:460] Webserver started at http://127.9.155.126:39261/ using document root <none> and password file <none>
I20260620 19:12:24.131974 22035 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:24.132025 22035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:12:24.132287 22035 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 19:12:24 dist-test-slave-n3fx krb5kdc[21591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982744, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.126@KRBTEST.COM for HTTP/127.9.155.84@KRBTEST.COM
I20260620 19:12:24.193917 22035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/master-0/data/instance:
uuid: "b7b289d31ad8472aa2f12a3684ec933e"
format_stamp: "Formatted at 2026-06-20 19:12:24 on dist-test-slave-n3fx"
server_key: "b5cb4998e254fab4ddfdfed83573eba1"
server_key_iv: "3d604a7878e45e0e9be3042674f56653"
server_key_version: "kuduclusterkey@0"
I20260620 19:12:24.194682 22035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/master-0/wal/instance:
uuid: "b7b289d31ad8472aa2f12a3684ec933e"
format_stamp: "Formatted at 2026-06-20 19:12:24 on dist-test-slave-n3fx"
server_key: "b5cb4998e254fab4ddfdfed83573eba1"
server_key_iv: "3d604a7878e45e0e9be3042674f56653"
server_key_version: "kuduclusterkey@0"
I20260620 19:12:24.199854 22035 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260620 19:12:24.217909 22050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:24.219480 22035 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 19:12:24.219633 22035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/master-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/master-0/wal
uuid: "b7b289d31ad8472aa2f12a3684ec933e"
format_stamp: "Formatted at 2026-06-20 19:12:24 on dist-test-slave-n3fx"
server_key: "b5cb4998e254fab4ddfdfed83573eba1"
server_key_iv: "3d604a7878e45e0e9be3042674f56653"
server_key_version: "kuduclusterkey@0"
I20260620 19:12:24.219775 22035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:24.234810 22035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:24.238111 22035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:24.238299 22035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:24.247215 22035 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.126:41439
I20260620 19:12:24.247226 22102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.126:41439 every 8 connection(s)
I20260620 19:12:24.248499 22035 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
I20260620 19:12:24.251372 22103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:12:24.255064 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 22035
I20260620 19:12:24.255195 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/master-0/wal/instance
I20260620 19:12:24.255481 9837 external_mini_cluster.cc:1428] Setting key 9fe163b2c87ed09ef7d7d4f21f59c18b
I20260620 19:12:24.257437 22103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7b289d31ad8472aa2f12a3684ec933e: Bootstrap starting.
I20260620 19:12:24.259765 22103 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7b289d31ad8472aa2f12a3684ec933e: Neither blocks nor log segments found. Creating new log.
I20260620 19:12:24.260632 22103 log.cc:824] T 00000000000000000000000000000000 P b7b289d31ad8472aa2f12a3684ec933e: Log is configured to *not* fsync() on all Append() calls
I20260620 19:12:24.262642 22103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7b289d31ad8472aa2f12a3684ec933e: No bootstrap required, opened a new log
Jun 20 19:12:24 dist-test-slave-n3fx krb5kdc[21591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:12:24.266785 22103 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7b289d31ad8472aa2f12a3684ec933e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7b289d31ad8472aa2f12a3684ec933e" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 41439 } }
I20260620 19:12:24.267081 22103 raft_consensus.cc:374] T 00000000000000000000000000000000 P b7b289d31ad8472aa2f12a3684ec933e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:12:24.267166 22103 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7b289d31ad8472aa2f12a3684ec933e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7b289d31ad8472aa2f12a3684ec933e, State: Initialized, Role: FOLLOWER
I20260620 19:12:24.267735 22103 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7b289d31ad8472aa2f12a3684ec933e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7b289d31ad8472aa2f12a3684ec933e" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 41439 } }
I20260620 19:12:24.267891 22103 raft_consensus.cc:388] T 00000000000000000000000000000000 P b7b289d31ad8472aa2f12a3684ec933e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:12:24.267971 22103 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7b289d31ad8472aa2f12a3684ec933e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:12:24.268103 22103 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7b289d31ad8472aa2f12a3684ec933e [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:12:24.269222 22103 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7b289d31ad8472aa2f12a3684ec933e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7b289d31ad8472aa2f12a3684ec933e" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 41439 } }
I20260620 19:12:24.269553 22103 leader_election.cc:302] T 00000000000000000000000000000000 P b7b289d31ad8472aa2f12a3684ec933e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b7b289d31ad8472aa2f12a3684ec933e; no voters:
I20260620 19:12:24.269953 22103 leader_election.cc:288] T 00000000000000000000000000000000 P b7b289d31ad8472aa2f12a3684ec933e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:12:24.270015 22108 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b7b289d31ad8472aa2f12a3684ec933e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:12:24.270253 22108 raft_consensus.cc:686] T 00000000000000000000000000000000 P b7b289d31ad8472aa2f12a3684ec933e [term 1 LEADER]: Becoming Leader. State: Replica: b7b289d31ad8472aa2f12a3684ec933e, State: Running, Role: LEADER
I20260620 19:12:24.270603 22108 consensus_queue.cc:237] T 00000000000000000000000000000000 P b7b289d31ad8472aa2f12a3684ec933e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7b289d31ad8472aa2f12a3684ec933e" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 41439 } }
I20260620 19:12:24.271306 22103 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7b289d31ad8472aa2f12a3684ec933e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:12:24.271831 22108 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7b289d31ad8472aa2f12a3684ec933e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7b289d31ad8472aa2f12a3684ec933e. Latest consensus state: current_term: 1 leader_uuid: "b7b289d31ad8472aa2f12a3684ec933e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7b289d31ad8472aa2f12a3684ec933e" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 41439 } } }
I20260620 19:12:24.271968 22108 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7b289d31ad8472aa2f12a3684ec933e [sys.catalog]: This master's current role is: LEADER
I20260620 19:12:24.272873 22106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:24.257003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40564 (local address 127.9.155.126:41439)
0620 19:12:24.257494 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:24.257513 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:24.257558 (+ 45us) server_negotiation.cc:407] Connection header received
0620 19:12:24.258466 (+ 908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:24.258493 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:24.258942 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:24.259329 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:24.260205 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:24.261839 (+ 1634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:24.263015 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:24.263739 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:24.263901 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:24.266885 (+ 2984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:24.266892 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:24.266911 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:24.266941 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:24.268867 (+ 1926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:24.269395 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:24.269402 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:24.269408 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:24.269546 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:24.269845 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:24.269848 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:24.269850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:24.270473 (+ 623us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:24.271384 (+ 911us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:24.271537 (+ 153us) server_negotiation.cc:299] Negotiation successful
0620 19:12:24.271817 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":154,"threads_started":1}
I20260620 19:12:24.273665 22103 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 19:12:24.272187 22108 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7b289d31ad8472aa2f12a3684ec933e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b7b289d31ad8472aa2f12a3684ec933e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7b289d31ad8472aa2f12a3684ec933e" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 41439 } } }
I20260620 19:12:24.274289 22108 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7b289d31ad8472aa2f12a3684ec933e [sys.catalog]: This master's current role is: LEADER
I20260620 19:12:24.277050 22112 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:12:24.281564 22112 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:12:24.288316 22112 catalog_manager.cc:1347] Generated new cluster ID: eee6e250b55643e1bba6ad6d0c8f29d5
I20260620 19:12:24.288383 22112 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:12:24.300163 22112 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:12:24.301117 22112 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:12:24.309561 22112 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b7b289d31ad8472aa2f12a3684ec933e: Generated new TSK 0
I20260620 19:12:24.310305 22112 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.155.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.155.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.155.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.65.keytab.
Entry for principal kudu/127.9.155.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.65.keytab.
I20260620 19:12:25.008057 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.155.84:58483/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.65:0
--local_ip_for_outbound_sockets=127.9.155.65
--webserver_interface=127.9.155.65
--webserver_port=0
--tserver_master_addrs=127.9.155.126:41439
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.65.keytab
--principal=kudu/127.9.155.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:41329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:12:25.125387 22148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:25.125775 22148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:25.125897 22148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:25.130468 22148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:25.130540 22148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:25.130625 22148 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.65
I20260620 19:12:25.136431 22148 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:41329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.9.155.84:58483/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.65.keytab
--principal=kudu/127.9.155.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.155.65
--webserver_port=0
--tserver_master_addrs=127.9.155.126:41439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22148
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.65
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:12:25.137512 22148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:25.138535 22148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:12:25.148453 22154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:25.149230 22153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:25.149441 22148 server_base.cc:1034] running on GCE node
W20260620 19:12:25.149608 22156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:25.150095 22148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:25.150869 22148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:25.152024 22148 hybrid_clock.cc:648] HybridClock initialized: now 1781982745151957 us; error 87 us; skew 500 ppm
Jun 20 19:12:25 dist-test-slave-n3fx krb5kdc[21591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982745, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:25.155476 22148 init.cc:377] Logged in from keytab as kudu/127.9.155.65@KRBTEST.COM (short username kudu)
I20260620 19:12:25.156783 22148 webserver.cc:460] Webserver started at http://127.9.155.65:34263/ using document root <none> and password file <none>
I20260620 19:12:25.157351 22148 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:25.157421 22148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:12:25.157642 22148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 19:12:25 dist-test-slave-n3fx krb5kdc[21591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982745, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.65@KRBTEST.COM for HTTP/127.9.155.84@KRBTEST.COM
I20260620 19:12:25.173882 22148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-0/data/instance:
uuid: "2b1c4d9b640440c48b207fd17ca3613e"
format_stamp: "Formatted at 2026-06-20 19:12:25 on dist-test-slave-n3fx"
server_key: "0d30afa62542cc0d75f1b6ed2182136a"
server_key_iv: "27d31db1c975e9bcf8679c1b2afb3d0b"
server_key_version: "kuduclusterkey@0"
I20260620 19:12:25.174743 22148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-0/wal/instance:
uuid: "2b1c4d9b640440c48b207fd17ca3613e"
format_stamp: "Formatted at 2026-06-20 19:12:25 on dist-test-slave-n3fx"
server_key: "0d30afa62542cc0d75f1b6ed2182136a"
server_key_iv: "27d31db1c975e9bcf8679c1b2afb3d0b"
server_key_version: "kuduclusterkey@0"
I20260620 19:12:25.181850 22148 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.003s
I20260620 19:12:25.199678 22165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:25.201582 22148 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260620 19:12:25.201745 22148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-0/wal
uuid: "2b1c4d9b640440c48b207fd17ca3613e"
format_stamp: "Formatted at 2026-06-20 19:12:25 on dist-test-slave-n3fx"
server_key: "0d30afa62542cc0d75f1b6ed2182136a"
server_key_iv: "27d31db1c975e9bcf8679c1b2afb3d0b"
server_key_version: "kuduclusterkey@0"
I20260620 19:12:25.201908 22148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:25.218516 22148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:25.222223 22148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:25.222481 22148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:25.223199 22148 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:12:25.224792 22148 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:12:25.224855 22148 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:25.224911 22148 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:12:25.224982 22148 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:25.235364 22148 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.65:34117
I20260620 19:12:25.235419 22278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.65:34117 every 8 connection(s)
I20260620 19:12:25.236347 22148 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
I20260620 19:12:25.242393 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 22148
I20260620 19:12:25.242511 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-0/wal/instance
I20260620 19:12:25.242792 9837 external_mini_cluster.cc:1428] Setting key 271a858c0f68e6275fdb9cc70ba83940
Jun 20 19:12:25 dist-test-slave-n3fx krb5kdc[21591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982745, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.65@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:12:25.259624 22282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:25.243541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:41439 (local address 127.9.155.65:36451)
0620 19:12:25.244925 (+ 1384us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:25.244982 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:25.246003 (+ 1021us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:25.246823 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:25.246833 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:25.247182 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:25.247845 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:25.247865 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:25.249051 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:25.249055 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:25.250432 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:25.250443 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:25.251210 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:25.251213 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:25.251396 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:25.252356 (+ 960us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:12:25.252377 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:25.254080 (+ 1703us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:25.256333 (+ 2253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:25.256339 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:25.256356 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:25.256712 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:25.257038 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:25.257041 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:25.257043 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:25.257223 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:25.257723 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:25.257730 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:25.258040 (+ 310us) client_negotiation.cc:769] Sending connection context
0620 19:12:25.258241 (+ 201us) client_negotiation.cc:240] Negotiation successful
0620 19:12:25.258575 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1164,"thread_start_us":130,"threads_started":1}
I20260620 19:12:25.260612 22281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:25.242619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.65:36451 (local address 127.9.155.126:41439)
0620 19:12:25.244665 (+ 2046us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:25.244674 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:25.246297 (+ 1623us) server_negotiation.cc:407] Connection header received
0620 19:12:25.246341 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:25.246345 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:25.246393 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:25.246495 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:25.248014 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:25.248907 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:25.250566 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:25.250936 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:25.251033 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:25.254267 (+ 3234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:25.254271 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:25.254275 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:25.254301 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:25.256178 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:25.256827 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:25.256831 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:25.256833 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:25.256922 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:25.257343 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:25.257346 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:25.257348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:25.257577 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:25.257669 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:25.260310 (+ 2641us) server_negotiation.cc:299] Negotiation successful
0620 19:12:25.260446 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1948,"thread_start_us":119,"threads_started":1}
WARNING: no policy specified for kudu/127.9.155.66@KRBTEST.COM; defaulting to no policy
I20260620 19:12:25.268016 22279 heartbeater.cc:347] Connected to a master server at 127.9.155.126:41439
I20260620 19:12:25.268335 22279 heartbeater.cc:464] Registering TS with master...
I20260620 19:12:25.268989 22279 heartbeater.cc:511] Master 127.9.155.126:41439 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.155.66@KRBTEST.COM" created.
I20260620 19:12:25.270967 22067 ts_manager.cc:194] Registered new tserver with Master: 2b1c4d9b640440c48b207fd17ca3613e (127.9.155.65:34117)
I20260620 19:12:25.272209 22067 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.155.65@KRBTEST.COM'} at 127.9.155.65:36451
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.155.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.66.keytab.
Entry for principal kudu/127.9.155.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.66.keytab.
I20260620 19:12:25.285873 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.155.84:58483/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.66:0
--local_ip_for_outbound_sockets=127.9.155.66
--webserver_interface=127.9.155.66
--webserver_port=0
--tserver_master_addrs=127.9.155.126:41439
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.66.keytab
--principal=kudu/127.9.155.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:41329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 19:12:25 dist-test-slave-n3fx krb5kdc[21591](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781982745, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 19:12:25.449579 22287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:25.449976 22287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:25.450038 22287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:25.456328 22287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:25.456460 22287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:25.456565 22287 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.66
I20260620 19:12:25.464600 22287 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:41329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.9.155.84:58483/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.66.keytab
--principal=kudu/127.9.155.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.155.66
--webserver_port=0
--tserver_master_addrs=127.9.155.126:41439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22287
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.66
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:12:25.466168 22287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:25.467514 22287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:12:25.480204 22296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:25.480609 22287 server_base.cc:1034] running on GCE node
W20260620 19:12:25.480242 22294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:25.485231 22293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:25.485924 22287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:25.486840 22287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:25.490144 22287 hybrid_clock.cc:648] HybridClock initialized: now 1781982745490032 us; error 115 us; skew 500 ppm
Jun 20 19:12:25 dist-test-slave-n3fx krb5kdc[21591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982745, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:25.494096 22287 init.cc:377] Logged in from keytab as kudu/127.9.155.66@KRBTEST.COM (short username kudu)
I20260620 19:12:25.495353 22287 webserver.cc:460] Webserver started at http://127.9.155.66:42817/ using document root <none> and password file <none>
I20260620 19:12:25.496142 22287 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:25.496273 22287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:12:25.496518 22287 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 19:12:25 dist-test-slave-n3fx krb5kdc[21591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982745, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.66@KRBTEST.COM for HTTP/127.9.155.84@KRBTEST.COM
I20260620 19:12:25.521523 22287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-1/data/instance:
uuid: "6cf1f2b0ca174b43b623f3189e587d85"
format_stamp: "Formatted at 2026-06-20 19:12:25 on dist-test-slave-n3fx"
server_key: "a846763720a94b5f60aa76e800230f98"
server_key_iv: "c4f7ab12f48b93a7818101f643f91b4d"
server_key_version: "kuduclusterkey@0"
I20260620 19:12:25.522341 22287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-1/wal/instance:
uuid: "6cf1f2b0ca174b43b623f3189e587d85"
format_stamp: "Formatted at 2026-06-20 19:12:25 on dist-test-slave-n3fx"
server_key: "a846763720a94b5f60aa76e800230f98"
server_key_iv: "c4f7ab12f48b93a7818101f643f91b4d"
server_key_version: "kuduclusterkey@0"
I20260620 19:12:25.527943 22287 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 19:12:25.559438 22303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:25.561224 22287 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 19:12:25.561362 22287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-1/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-1/wal
uuid: "6cf1f2b0ca174b43b623f3189e587d85"
format_stamp: "Formatted at 2026-06-20 19:12:25 on dist-test-slave-n3fx"
server_key: "a846763720a94b5f60aa76e800230f98"
server_key_iv: "c4f7ab12f48b93a7818101f643f91b4d"
server_key_version: "kuduclusterkey@0"
I20260620 19:12:25.561494 22287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:25.579749 22287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:25.582700 22287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:25.582871 22287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:25.583436 22287 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:12:25.584369 22287 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:12:25.584481 22287 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:25.584582 22287 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:12:25.584662 22287 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:25.595690 22287 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.66:42347
I20260620 19:12:25.595829 22416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.66:42347 every 8 connection(s)
I20260620 19:12:25.596951 22287 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
I20260620 19:12:25.599210 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 22287
I20260620 19:12:25.599326 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-1/wal/instance
I20260620 19:12:25.599659 9837 external_mini_cluster.cc:1428] Setting key 826c5c1d0a8361754a805cc22a0925b2
WARNING: no policy specified for kudu/127.9.155.67@KRBTEST.COM; defaulting to no policy
Jun 20 19:12:25 dist-test-slave-n3fx krb5kdc[21591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982745, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.66@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.155.67@KRBTEST.COM" created.
I20260620 19:12:25.634110 22281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:25.600725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.66:33447 (local address 127.9.155.126:41439)
0620 19:12:25.600868 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:25.600873 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:25.604834 (+ 3961us) server_negotiation.cc:407] Connection header received
0620 19:12:25.606175 (+ 1341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:25.606179 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:25.606244 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:25.606300 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:25.608113 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:25.608901 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:25.610535 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:25.610866 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:25.610967 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:25.627104 (+ 16137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:25.627109 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:25.627112 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:25.627146 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:25.628927 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:25.631799 (+ 2872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:25.631804 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:25.631806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:25.631916 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:25.632644 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:25.632649 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:25.632650 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:25.632903 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:25.632985 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:25.633771 (+ 786us) server_negotiation.cc:299] Negotiation successful
0620 19:12:25.633927 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 19:12:25.635923 22419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:25.599590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:41439 (local address 127.9.155.66:33447)
0620 19:12:25.604614 (+ 5024us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:25.604697 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:25.605864 (+ 1167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:25.606799 (+ 935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:25.606810 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:25.607261 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:25.607944 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:25.607964 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:25.609064 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:25.609067 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:25.610415 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:25.610425 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:25.621801 (+ 11376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:25.621808 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:25.622127 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:25.623127 (+ 1000us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:12:25.623155 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:25.626722 (+ 3567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:25.631134 (+ 4412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:25.631143 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:25.631166 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:25.631637 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:25.632259 (+ 622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:25.632262 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:25.632264 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:25.632515 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:25.633073 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:25.633081 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:25.633449 (+ 368us) client_negotiation.cc:769] Sending connection context
0620 19:12:25.634366 (+ 917us) client_negotiation.cc:240] Negotiation successful
0620 19:12:25.634752 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4721,"thread_start_us":144,"threads_started":1}
I20260620 19:12:25.637308 22417 heartbeater.cc:347] Connected to a master server at 127.9.155.126:41439
I20260620 19:12:25.637588 22417 heartbeater.cc:464] Registering TS with master...
I20260620 19:12:25.638190 22417 heartbeater.cc:511] Master 127.9.155.126:41439 requested a full tablet report, sending...
I20260620 19:12:25.639312 22067 ts_manager.cc:194] Registered new tserver with Master: 6cf1f2b0ca174b43b623f3189e587d85 (127.9.155.66:42347)
I20260620 19:12:25.640095 22067 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.155.66@KRBTEST.COM'} at 127.9.155.66:33447
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.155.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.67.keytab.
Entry for principal kudu/127.9.155.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.67.keytab.
I20260620 19:12:25.674620 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.155.84:58483/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.67:0
--local_ip_for_outbound_sockets=127.9.155.67
--webserver_interface=127.9.155.67
--webserver_port=0
--tserver_master_addrs=127.9.155.126:41439
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.67.keytab
--principal=kudu/127.9.155.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:41329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 19:12:25 dist-test-slave-n3fx krb5kdc[21591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982745, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.126@KRBTEST.COM for HTTP/127.9.155.84@KRBTEST.COM
W20260620 19:12:25.849856 22422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:25.850356 22422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:25.850472 22422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:25.856742 22422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:25.856889 22422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:25.857023 22422 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.67
I20260620 19:12:25.865312 22422 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:41329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.9.155.84:58483/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.67.keytab
--principal=kudu/127.9.155.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.155.67
--webserver_port=0
--tserver_master_addrs=127.9.155.126:41439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22422
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.67
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:12:25.867255 22422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:25.868901 22422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:12:25.883644 22428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:25.884255 22427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:25.885161 22422 server_base.cc:1034] running on GCE node
W20260620 19:12:25.888366 22430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:25.889008 22422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:25.889883 22422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:25.921500 22422 hybrid_clock.cc:648] HybridClock initialized: now 1781982745921346 us; error 179 us; skew 500 ppm
Jun 20 19:12:25 dist-test-slave-n3fx krb5kdc[21591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982745, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:25.925683 22422 init.cc:377] Logged in from keytab as kudu/127.9.155.67@KRBTEST.COM (short username kudu)
I20260620 19:12:25.926986 22422 webserver.cc:460] Webserver started at http://127.9.155.67:44203/ using document root <none> and password file <none>
I20260620 19:12:25.927752 22422 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:25.927858 22422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:12:25.928118 22422 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 19:12:25 dist-test-slave-n3fx krb5kdc[21591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982745, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.67@KRBTEST.COM for HTTP/127.9.155.84@KRBTEST.COM
I20260620 19:12:25.952890 22422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-2/data/instance:
uuid: "416cb81100bc4f688bdba356c852a2e8"
format_stamp: "Formatted at 2026-06-20 19:12:25 on dist-test-slave-n3fx"
server_key: "722e9de5174bdf6bd314112c9b40ed8a"
server_key_iv: "ff0ccc818171589316fa8b0eb505d22f"
server_key_version: "kuduclusterkey@0"
I20260620 19:12:25.953683 22422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-2/wal/instance:
uuid: "416cb81100bc4f688bdba356c852a2e8"
format_stamp: "Formatted at 2026-06-20 19:12:25 on dist-test-slave-n3fx"
server_key: "722e9de5174bdf6bd314112c9b40ed8a"
server_key_iv: "ff0ccc818171589316fa8b0eb505d22f"
server_key_version: "kuduclusterkey@0"
I20260620 19:12:25.966154 22422 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.005s sys 0.000s
I20260620 19:12:25.990089 22437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:25.991729 22422 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 19:12:25.991863 22422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-2/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-2/wal
uuid: "416cb81100bc4f688bdba356c852a2e8"
format_stamp: "Formatted at 2026-06-20 19:12:25 on dist-test-slave-n3fx"
server_key: "722e9de5174bdf6bd314112c9b40ed8a"
server_key_iv: "ff0ccc818171589316fa8b0eb505d22f"
server_key_version: "kuduclusterkey@0"
I20260620 19:12:25.992004 22422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:26.022367 22422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:26.026590 22422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:26.026863 22422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:26.027547 22422 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:12:26.028626 22422 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:12:26.028723 22422 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:26.028813 22422 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:12:26.028873 22422 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:26.040071 22422 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.67:39833
I20260620 19:12:26.041935 22422 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
I20260620 19:12:26.045073 22550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.67:39833 every 8 connection(s)
I20260620 19:12:26.047866 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 22422
I20260620 19:12:26.047979 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-2/wal/instance
I20260620 19:12:26.048267 9837 external_mini_cluster.cc:1428] Setting key 5804b7cf3d61f541f93e3b06b16ac7a0
Jun 20 19:12:26 dist-test-slave-n3fx krb5kdc[21591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982745, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.67@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:12:26.087078 22281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:26.048686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.67:39349 (local address 127.9.155.126:41439)
0620 19:12:26.049043 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:26.049047 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:26.057403 (+ 8356us) server_negotiation.cc:407] Connection header received
0620 19:12:26.060429 (+ 3026us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:26.060434 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:26.060504 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:26.060559 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:26.067074 (+ 6515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:26.067930 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:26.069824 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:26.070144 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:26.070221 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:26.080970 (+ 10749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:26.080975 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:26.080978 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:26.081012 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:26.082824 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:26.083553 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:26.083557 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:26.083559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:26.083661 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:26.084063 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:26.084066 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:26.084068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:26.084338 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:26.084420 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:26.086765 (+ 2345us) server_negotiation.cc:299] Negotiation successful
0620 19:12:26.086912 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269}
I20260620 19:12:26.089099 22553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:26.047420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:41439 (local address 127.9.155.67:39349)
0620 19:12:26.057157 (+ 9737us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:26.057255 (+ 98us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:26.058557 (+ 1302us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:26.062847 (+ 4290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:26.062859 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:26.063268 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:26.064048 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:26.064070 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:26.068144 (+ 4074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:26.068148 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:26.069671 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:26.069686 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:26.070450 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:26.070454 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:26.070678 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:26.074241 (+ 3563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:12:26.074265 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:26.080697 (+ 6432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:26.082999 (+ 2302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:26.083007 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:26.083029 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:26.083424 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:26.083765 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:26.083768 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:26.083770 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:26.083966 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:26.086095 (+ 2129us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:26.086102 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:26.086473 (+ 371us) client_negotiation.cc:769] Sending connection context
0620 19:12:26.087542 (+ 1069us) client_negotiation.cc:240] Negotiation successful
0620 19:12:26.087981 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9411,"thread_start_us":131,"threads_started":1}
I20260620 19:12:26.090763 22551 heartbeater.cc:347] Connected to a master server at 127.9.155.126:41439
I20260620 19:12:26.091017 22551 heartbeater.cc:464] Registering TS with master...
I20260620 19:12:26.091572 22551 heartbeater.cc:511] Master 127.9.155.126:41439 requested a full tablet report, sending...
I20260620 19:12:26.092742 22067 ts_manager.cc:194] Registered new tserver with Master: 416cb81100bc4f688bdba356c852a2e8 (127.9.155.67:39833)
I20260620 19:12:26.093493 22067 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.155.67@KRBTEST.COM'} at 127.9.155.67:39349
I20260620 19:12:26.097966 9837 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:12:26.098763 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 22148
I20260620 19:12:26.105931 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 22287
I20260620 19:12:26.113940 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 22422
I20260620 19:12:26.121542 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 22035
I20260620 19:12:26.129180 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:41439
--webserver_interface=127.9.155.126
--webserver_port=39261
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab
--principal=kudu/127.9.155.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:41329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.155.126:41439
--ranger_config_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.155.84:58483/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.9.155.84:58483/kms with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:12:26.246925 22554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:26.247210 22554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:26.247249 22554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:26.251417 22554 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:12:26.251487 22554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:26.251506 22554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:26.251521 22554 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:12:26.251535 22554 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:12:26.257086 22554 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:41329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.9.155.84:58483/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.9.155.126:41439
--ranger_config_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab
--principal=kudu/127.9.155.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:41439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.155.126
--webserver_port=39261
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22554
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:12:26.258121 22554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:26.259104 22554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 19:12:26.267742 22554 server_base.cc:1034] running on GCE node
W20260620 19:12:26.267704 22562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:26.267704 22559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:26.267722 22560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:26.268756 22554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:26.269929 22554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:26.271091 22554 hybrid_clock.cc:648] HybridClock initialized: now 1781982746271070 us; error 40 us; skew 500 ppm
Jun 20 19:12:26 dist-test-slave-n3fx krb5kdc[21591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982746, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:26.273840 22554 init.cc:377] Logged in from keytab as kudu/127.9.155.126@KRBTEST.COM (short username kudu)
I20260620 19:12:26.274704 22554 webserver.cc:460] Webserver started at http://127.9.155.126:39261/ using document root <none> and password file <none>
I20260620 19:12:26.275203 22554 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:26.275272 22554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 19:12:26 dist-test-slave-n3fx krb5kdc[21591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982746, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.126@KRBTEST.COM for HTTP/127.9.155.84@KRBTEST.COM
I20260620 19:12:26.327560 22554 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 19:12:26.329922 22570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:26.331252 22554 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 19:12:26.331377 22554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/master-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/master-0/wal
uuid: "b7b289d31ad8472aa2f12a3684ec933e"
format_stamp: "Formatted at 2026-06-20 19:12:24 on dist-test-slave-n3fx"
server_key: "b5cb4998e254fab4ddfdfed83573eba1"
server_key_iv: "3d604a7878e45e0e9be3042674f56653"
server_key_version: "kuduclusterkey@0"
I20260620 19:12:26.331940 22554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:26.356948 22554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:26.359833 22554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:26.360005 22554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:26.367153 22554 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.126:41439
I20260620 19:12:26.367187 22622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.126:41439 every 8 connection(s)
I20260620 19:12:26.368244 22554 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
I20260620 19:12:26.372519 22623 sys_catalog.cc:263] Verifying existing consensus state
I20260620 19:12:26.374419 22623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7b289d31ad8472aa2f12a3684ec933e: Bootstrap starting.
I20260620 19:12:26.377753 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 22554
I20260620 19:12:26.378140 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.155.84:58483/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.65:34117
--local_ip_for_outbound_sockets=127.9.155.65
--tserver_master_addrs=127.9.155.126:41439
--webserver_port=34263
--webserver_interface=127.9.155.65
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.65.keytab
--principal=kudu/127.9.155.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:41329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.9.155.84:58483/kms with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:12:26.390975 22623 log.cc:824] T 00000000000000000000000000000000 P b7b289d31ad8472aa2f12a3684ec933e: Log is configured to *not* fsync() on all Append() calls
I20260620 19:12:26.397864 22623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7b289d31ad8472aa2f12a3684ec933e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:12:26.398236 22623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7b289d31ad8472aa2f12a3684ec933e: Bootstrap complete.
I20260620 19:12:26.402279 22623 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7b289d31ad8472aa2f12a3684ec933e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7b289d31ad8472aa2f12a3684ec933e" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 41439 } }
I20260620 19:12:26.402675 22623 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7b289d31ad8472aa2f12a3684ec933e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7b289d31ad8472aa2f12a3684ec933e, State: Initialized, Role: FOLLOWER
I20260620 19:12:26.403139 22623 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7b289d31ad8472aa2f12a3684ec933e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7b289d31ad8472aa2f12a3684ec933e" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 41439 } }
I20260620 19:12:26.403258 22623 raft_consensus.cc:388] T 00000000000000000000000000000000 P b7b289d31ad8472aa2f12a3684ec933e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:12:26.403323 22623 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7b289d31ad8472aa2f12a3684ec933e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:12:26.403415 22623 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7b289d31ad8472aa2f12a3684ec933e [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:12:26.405365 22623 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7b289d31ad8472aa2f12a3684ec933e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7b289d31ad8472aa2f12a3684ec933e" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 41439 } }
I20260620 19:12:26.405670 22623 leader_election.cc:302] T 00000000000000000000000000000000 P b7b289d31ad8472aa2f12a3684ec933e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b7b289d31ad8472aa2f12a3684ec933e; no voters:
I20260620 19:12:26.405964 22623 leader_election.cc:288] T 00000000000000000000000000000000 P b7b289d31ad8472aa2f12a3684ec933e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 19:12:26.406065 22627 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b7b289d31ad8472aa2f12a3684ec933e [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:12:26.406265 22627 raft_consensus.cc:686] T 00000000000000000000000000000000 P b7b289d31ad8472aa2f12a3684ec933e [term 2 LEADER]: Becoming Leader. State: Replica: b7b289d31ad8472aa2f12a3684ec933e, State: Running, Role: LEADER
I20260620 19:12:26.406647 22627 consensus_queue.cc:237] T 00000000000000000000000000000000 P b7b289d31ad8472aa2f12a3684ec933e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7b289d31ad8472aa2f12a3684ec933e" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 41439 } }
I20260620 19:12:26.407244 22623 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7b289d31ad8472aa2f12a3684ec933e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:12:26.408106 22629 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7b289d31ad8472aa2f12a3684ec933e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7b289d31ad8472aa2f12a3684ec933e. Latest consensus state: current_term: 2 leader_uuid: "b7b289d31ad8472aa2f12a3684ec933e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7b289d31ad8472aa2f12a3684ec933e" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 41439 } } }
I20260620 19:12:26.408113 22628 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7b289d31ad8472aa2f12a3684ec933e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b7b289d31ad8472aa2f12a3684ec933e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7b289d31ad8472aa2f12a3684ec933e" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 41439 } } }
I20260620 19:12:26.408365 22628 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7b289d31ad8472aa2f12a3684ec933e [sys.catalog]: This master's current role is: LEADER
I20260620 19:12:26.408648 22629 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7b289d31ad8472aa2f12a3684ec933e [sys.catalog]: This master's current role is: LEADER
I20260620 19:12:26.409291 22636 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:12:26.409601 22623 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 19:12:26.415045 22636 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:12:26.415792 22636 catalog_manager.cc:1259] Loaded cluster ID: eee6e250b55643e1bba6ad6d0c8f29d5
I20260620 19:12:26.415897 22636 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:12:26.418542 22636 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:12:26.419318 22636 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b7b289d31ad8472aa2f12a3684ec933e: Loaded TSK: 0
I20260620 19:12:26.420656 22636 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 19:12:26.506227 22625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:26.506502 22625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:26.506567 22625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:26.511755 22625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:26.511969 22625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:26.512117 22625 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.65
I20260620 19:12:26.519025 22625 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:41329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.9.155.84:58483/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.65.keytab
--principal=kudu/127.9.155.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.65:34117
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.155.65
--webserver_port=34263
--tserver_master_addrs=127.9.155.126:41439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22625
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.65
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:12:26.520469 22625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:26.521485 22625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:12:26.531361 22661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:26.531785 22625 server_base.cc:1034] running on GCE node
W20260620 19:12:26.532068 22663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:26.532347 22660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:26.532886 22625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:26.533882 22625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:26.535072 22625 hybrid_clock.cc:648] HybridClock initialized: now 1781982746535043 us; error 49 us; skew 500 ppm
Jun 20 19:12:26 dist-test-slave-n3fx krb5kdc[21591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982746, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:26.538137 22625 init.cc:377] Logged in from keytab as kudu/127.9.155.65@KRBTEST.COM (short username kudu)
I20260620 19:12:26.539129 22625 webserver.cc:460] Webserver started at http://127.9.155.65:34263/ using document root <none> and password file <none>
I20260620 19:12:26.539827 22625 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:26.540024 22625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 19:12:26 dist-test-slave-n3fx krb5kdc[21591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982746, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.65@KRBTEST.COM for HTTP/127.9.155.84@KRBTEST.COM
I20260620 19:12:26.560453 22625 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:12:26.569350 22671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:26.571152 22625 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.002s sys 0.000s
I20260620 19:12:26.571358 22625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-0/wal
uuid: "2b1c4d9b640440c48b207fd17ca3613e"
format_stamp: "Formatted at 2026-06-20 19:12:25 on dist-test-slave-n3fx"
server_key: "0d30afa62542cc0d75f1b6ed2182136a"
server_key_iv: "27d31db1c975e9bcf8679c1b2afb3d0b"
server_key_version: "kuduclusterkey@0"
I20260620 19:12:26.572152 22625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:26.585434 22625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:26.588361 22625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:26.588533 22625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:26.589108 22625 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:12:26.589978 22625 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:12:26.590065 22625 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:26.590153 22625 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:12:26.590201 22625 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:26.600096 22625 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.65:34117
I20260620 19:12:26.600332 22784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.65:34117 every 8 connection(s)
I20260620 19:12:26.601431 22625 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
I20260620 19:12:26.608134 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 22625
I20260620 19:12:26.608675 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.155.84:58483/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.66:42347
--local_ip_for_outbound_sockets=127.9.155.66
--tserver_master_addrs=127.9.155.126:41439
--webserver_port=42817
--webserver_interface=127.9.155.66
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.66.keytab
--principal=kudu/127.9.155.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:41329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.9.155.84:58483/kms with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 19:12:26 dist-test-slave-n3fx krb5kdc[21591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982746, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.65@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:12:26.622752 22787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:26.603774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:41439 (local address 127.9.155.65:50815)
0620 19:12:26.604309 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:26.604360 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:26.605283 (+ 923us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:26.608972 (+ 3689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:26.608980 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:26.609311 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:26.609933 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:26.609955 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:26.611722 (+ 1767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:26.611725 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:26.612778 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:26.612788 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:26.614069 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:26.614071 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:26.614213 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:26.614763 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:12:26.614784 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:26.616574 (+ 1790us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:26.619076 (+ 2502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:26.619083 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:26.619098 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:26.619421 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:26.619841 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:26.619844 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:26.619846 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:26.620021 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:26.620837 (+ 816us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:26.620844 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:26.621140 (+ 296us) client_negotiation.cc:769] Sending connection context
0620 19:12:26.621333 (+ 193us) client_negotiation.cc:240] Negotiation successful
0620 19:12:26.621686 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":116,"threads_started":1}
I20260620 19:12:26.624701 22788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:26.604387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.65:50815 (local address 127.9.155.126:41439)
0620 19:12:26.605005 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:26.605025 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:26.605072 (+ 47us) server_negotiation.cc:407] Connection header received
0620 19:12:26.607980 (+ 2908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:26.607998 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:26.608468 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:26.608873 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:26.610141 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:26.611548 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:26.612890 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:26.613692 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:26.613852 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:26.616811 (+ 2959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:26.616818 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:26.616835 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:26.616869 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:26.618932 (+ 2063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:26.619557 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:26.619564 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:26.619569 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:26.619727 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:26.620134 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:26.620138 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:26.620139 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:26.620664 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:26.620786 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:26.623350 (+ 2564us) server_negotiation.cc:299] Negotiation successful
0620 19:12:26.623680 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":148,"threads_started":1}
I20260620 19:12:26.628484 22785 heartbeater.cc:347] Connected to a master server at 127.9.155.126:41439
I20260620 19:12:26.628749 22785 heartbeater.cc:464] Registering TS with master...
I20260620 19:12:26.629310 22785 heartbeater.cc:511] Master 127.9.155.126:41439 requested a full tablet report, sending...
W20260620 19:12:26.631601 22785 heartbeater.cc:644] Failed to heartbeat to 127.9.155.126:41439 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 19:12:26.631731 22785 heartbeater.cc:464] Registering TS with master...
I20260620 19:12:26.631920 22785 heartbeater.cc:511] Master 127.9.155.126:41439 requested a full tablet report, sending...
I20260620 19:12:26.632803 22785 heartbeater.cc:464] Registering TS with master...
I20260620 19:12:26.633050 22785 heartbeater.cc:511] Master 127.9.155.126:41439 requested a full tablet report, sending...
W20260620 19:12:26.633857 22785 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 19:12:26.747889 22789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:26.748309 22789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:26.748436 22789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:26.754673 22789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:26.754768 22789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:26.754858 22789 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.66
I20260620 19:12:26.763303 22789 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:41329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.9.155.84:58483/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.66.keytab
--principal=kudu/127.9.155.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.66:42347
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.155.66
--webserver_port=42817
--tserver_master_addrs=127.9.155.126:41439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22789
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.66
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:12:26.764521 22789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:26.765620 22789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:12:26.776165 22794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:26.776173 22795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:26.776605 22797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:26.776827 22789 server_base.cc:1034] running on GCE node
I20260620 19:12:26.777230 22789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:26.777881 22789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:26.779070 22789 hybrid_clock.cc:648] HybridClock initialized: now 1781982746779031 us; error 48 us; skew 500 ppm
Jun 20 19:12:26 dist-test-slave-n3fx krb5kdc[21591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982746, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:26.782608 22789 init.cc:377] Logged in from keytab as kudu/127.9.155.66@KRBTEST.COM (short username kudu)
I20260620 19:12:26.783743 22789 webserver.cc:460] Webserver started at http://127.9.155.66:42817/ using document root <none> and password file <none>
I20260620 19:12:26.784430 22789 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:26.784507 22789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 19:12:26 dist-test-slave-n3fx krb5kdc[21591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982746, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.66@KRBTEST.COM for HTTP/127.9.155.84@KRBTEST.COM
I20260620 19:12:26.805583 22789 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.003s sys 0.000s
I20260620 19:12:26.808741 22805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:26.810365 22789 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260620 19:12:26.810503 22789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-1/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-1/wal
uuid: "6cf1f2b0ca174b43b623f3189e587d85"
format_stamp: "Formatted at 2026-06-20 19:12:25 on dist-test-slave-n3fx"
server_key: "a846763720a94b5f60aa76e800230f98"
server_key_iv: "c4f7ab12f48b93a7818101f643f91b4d"
server_key_version: "kuduclusterkey@0"
I20260620 19:12:26.811242 22789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:26.834157 22789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:26.837925 22789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:26.838191 22789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:26.838908 22789 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:12:26.839972 22789 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:12:26.840049 22789 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:26.840106 22789 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:12:26.840138 22789 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:26.850831 22789 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.66:42347
I20260620 19:12:26.852291 22789 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
I20260620 19:12:26.855269 22918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.66:42347 every 8 connection(s)
I20260620 19:12:26.859871 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 22789
I20260620 19:12:26.860417 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.155.84:58483/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.67:39833
--local_ip_for_outbound_sockets=127.9.155.67
--tserver_master_addrs=127.9.155.126:41439
--webserver_port=44203
--webserver_interface=127.9.155.67
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.67.keytab
--principal=kudu/127.9.155.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:41329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.9.155.84:58483/kms with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 19:12:26 dist-test-slave-n3fx krb5kdc[21591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982746, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.66@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:12:26.871979 22788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:26.858402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.66:52277 (local address 127.9.155.126:41439)
0620 19:12:26.858533 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:26.858538 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:26.858554 (+ 16us) server_negotiation.cc:407] Connection header received
0620 19:12:26.859674 (+ 1120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:26.859678 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:26.859738 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:26.859793 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:26.861475 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:26.862306 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:26.864001 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:26.864403 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:26.864512 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:26.867607 (+ 3095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:26.867612 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:26.867615 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:26.867647 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:26.869420 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:26.870109 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:26.870113 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:26.870115 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:26.870212 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:26.870632 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:26.870635 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:26.870637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:26.870868 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:26.870958 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:26.871656 (+ 698us) server_negotiation.cc:299] Negotiation successful
0620 19:12:26.871797 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 19:12:26.873929 22921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:26.857308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:41439 (local address 127.9.155.66:52277)
0620 19:12:26.857853 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:26.857905 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:26.859422 (+ 1517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:26.860258 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:26.860267 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:26.860624 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:26.861293 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:26.861313 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:26.862464 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:26.862467 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:26.863857 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:26.863868 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:26.864698 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:26.864701 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:26.864913 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:26.865529 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:12:26.865548 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:26.867364 (+ 1816us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:26.869587 (+ 2223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:26.869594 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:26.869615 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:26.869983 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:26.870325 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:26.870328 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:26.870330 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:26.870523 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:26.871014 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:26.871022 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:26.871374 (+ 352us) client_negotiation.cc:769] Sending connection context
0620 19:12:26.872387 (+ 1013us) client_negotiation.cc:240] Negotiation successful
0620 19:12:26.872781 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":158,"threads_started":1}
I20260620 19:12:26.875566 22919 heartbeater.cc:347] Connected to a master server at 127.9.155.126:41439
I20260620 19:12:26.875833 22919 heartbeater.cc:464] Registering TS with master...
I20260620 19:12:26.876493 22919 heartbeater.cc:511] Master 127.9.155.126:41439 requested a full tablet report, sending...
W20260620 19:12:26.878427 22919 heartbeater.cc:644] Failed to heartbeat to 127.9.155.126:41439 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 19:12:26.878559 22919 heartbeater.cc:464] Registering TS with master...
I20260620 19:12:26.878754 22919 heartbeater.cc:511] Master 127.9.155.126:41439 requested a full tablet report, sending...
I20260620 19:12:26.879712 22919 heartbeater.cc:464] Registering TS with master...
I20260620 19:12:26.879889 22919 heartbeater.cc:511] Master 127.9.155.126:41439 requested a full tablet report, sending...
W20260620 19:12:26.880738 22919 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 19:12:27.021147 22922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:27.021651 22922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:27.021786 22922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:27.027947 22922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:27.028084 22922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:27.028228 22922 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.67
I20260620 19:12:27.036340 22922 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:41329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.9.155.84:58483/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.67.keytab
--principal=kudu/127.9.155.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.67:39833
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.155.67
--webserver_port=44203
--tserver_master_addrs=127.9.155.126:41439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22922
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.67
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:12:27.037832 22922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:27.039259 22922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:12:27.050935 22928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:27.050952 22929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:27.051384 22931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:27.051540 22922 server_base.cc:1034] running on GCE node
I20260620 19:12:27.051964 22922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:27.052708 22922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:27.053920 22922 hybrid_clock.cc:648] HybridClock initialized: now 1781982747053889 us; error 43 us; skew 500 ppm
Jun 20 19:12:27 dist-test-slave-n3fx krb5kdc[21591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982747, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:27.058187 22922 init.cc:377] Logged in from keytab as kudu/127.9.155.67@KRBTEST.COM (short username kudu)
I20260620 19:12:27.059484 22922 webserver.cc:460] Webserver started at http://127.9.155.67:44203/ using document root <none> and password file <none>
I20260620 19:12:27.060209 22922 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:27.060346 22922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 19:12:27 dist-test-slave-n3fx krb5kdc[21591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982747, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.67@KRBTEST.COM for HTTP/127.9.155.84@KRBTEST.COM
I20260620 19:12:27.101006 22922 fs_manager.cc:709] Time spent opening directory manager: real 0.017s user 0.000s sys 0.003s
I20260620 19:12:27.103760 22939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:27.105212 22922 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 19:12:27.105324 22922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-2/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-2/wal
uuid: "416cb81100bc4f688bdba356c852a2e8"
format_stamp: "Formatted at 2026-06-20 19:12:25 on dist-test-slave-n3fx"
server_key: "722e9de5174bdf6bd314112c9b40ed8a"
server_key_iv: "ff0ccc818171589316fa8b0eb505d22f"
server_key_version: "kuduclusterkey@0"
I20260620 19:12:27.105896 22922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:27.119545 22922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:27.123188 22922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:27.123413 22922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:27.124006 22922 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:12:27.124922 22922 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:12:27.124970 22922 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:27.125010 22922 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:12:27.125034 22922 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:27.135553 22922 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.67:39833
I20260620 19:12:27.135603 23052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.67:39833 every 8 connection(s)
I20260620 19:12:27.137391 22922 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
I20260620 19:12:27.147464 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 22922
Jun 20 19:12:27 dist-test-slave-n3fx krb5kdc[21591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982747, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.67@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:12:27.156293 22788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:27.139493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.67:34755 (local address 127.9.155.126:41439)
0620 19:12:27.139641 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:27.139646 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:27.143582 (+ 3936us) server_negotiation.cc:407] Connection header received
0620 19:12:27.144748 (+ 1166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:27.144752 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:27.144812 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:27.144917 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:27.146261 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:27.147110 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:27.149745 (+ 2635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:27.149969 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:27.150055 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:27.152527 (+ 2472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:27.152530 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:27.152532 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:27.152557 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:27.153863 (+ 1306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:27.154470 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:27.154474 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:27.154476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:27.154587 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:27.154930 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:27.154933 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:27.154934 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:27.155134 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:27.155236 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:27.155952 (+ 716us) server_negotiation.cc:299] Negotiation successful
0620 19:12:27.156079 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 19:12:27.157918 23055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:27.140007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:41439 (local address 127.9.155.67:34755)
0620 19:12:27.143390 (+ 3383us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:27.143446 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:27.144480 (+ 1034us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:27.145063 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:27.145070 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:27.145343 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:27.146073 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:27.146094 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:27.148393 (+ 2299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:27.148397 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:27.149628 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:27.149646 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:27.150108 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:27.150113 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:27.150270 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:27.150937 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:12:27.150958 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:27.152337 (+ 1379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:27.153972 (+ 1635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:27.153979 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:27.153998 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:27.154336 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:27.154684 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:27.154686 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:27.154688 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:27.154829 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:27.155295 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:27.155302 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:27.155637 (+ 335us) client_negotiation.cc:769] Sending connection context
0620 19:12:27.156810 (+ 1173us) client_negotiation.cc:240] Negotiation successful
0620 19:12:27.157132 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3152,"thread_start_us":2969,"threads_started":1}
I20260620 19:12:27.159483 23053 heartbeater.cc:347] Connected to a master server at 127.9.155.126:41439
I20260620 19:12:27.159734 23053 heartbeater.cc:464] Registering TS with master...
I20260620 19:12:27.160241 23053 heartbeater.cc:511] Master 127.9.155.126:41439 requested a full tablet report, sending...
W20260620 19:12:27.161939 23053 heartbeater.cc:644] Failed to heartbeat to 127.9.155.126:41439 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 19:12:27.162057 23053 heartbeater.cc:464] Registering TS with master...
I20260620 19:12:27.162225 23053 heartbeater.cc:511] Master 127.9.155.126:41439 requested a full tablet report, sending...
I20260620 19:12:27.162468 23056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:27.148188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40578 (local address 127.9.155.126:41439)
0620 19:12:27.148713 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:27.148717 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:27.148916 (+ 199us) server_negotiation.cc:407] Connection header received
0620 19:12:27.149054 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:27.149058 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:27.149109 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:27.149206 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:27.150409 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:27.151132 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:27.154037 (+ 2905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:27.154385 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:27.154466 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:27.156532 (+ 2066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:27.156535 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:27.156538 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:27.156560 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:27.159369 (+ 2809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:27.160674 (+ 1305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:27.160678 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:27.160679 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:27.160763 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:27.161077 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:27.161081 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:27.161082 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:27.161327 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:27.161521 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:27.162183 (+ 662us) server_negotiation.cc:299] Negotiation successful
0620 19:12:27.162306 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":84,"threads_started":1}
I20260620 19:12:27.163571 23053 heartbeater.cc:464] Registering TS with master...
I20260620 19:12:27.163749 23053 heartbeater.cc:511] Master 127.9.155.126:41439 requested a full tablet report, sending...
W20260620 19:12:27.164551 23053 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 20 19:12:27 dist-test-slave-n3fx krb5kdc[21591](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781982747, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:27.634152 22785 heartbeater.cc:464] Registering TS with master...
I20260620 19:12:27.634516 22785 heartbeater.cc:511] Master 127.9.155.126:41439 requested a full tablet report, sending...
I20260620 19:12:27.636019 22586 ts_manager.cc:194] Registered new tserver with Master: 2b1c4d9b640440c48b207fd17ca3613e (127.9.155.65:34117)
I20260620 19:12:27.637779 22586 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.155.65@KRBTEST.COM'} at 127.9.155.65:50815
I20260620 19:12:27.880959 22919 heartbeater.cc:464] Registering TS with master...
I20260620 19:12:27.881269 22919 heartbeater.cc:511] Master 127.9.155.126:41439 requested a full tablet report, sending...
I20260620 19:12:27.881994 22586 ts_manager.cc:194] Registered new tserver with Master: 6cf1f2b0ca174b43b623f3189e587d85 (127.9.155.66:42347)
I20260620 19:12:27.882730 22586 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.155.66@KRBTEST.COM'} at 127.9.155.66:52277
Jun 20 19:12:27 dist-test-slave-n3fx krb5kdc[21591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982747, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.126@KRBTEST.COM for HTTP/127.9.155.84@KRBTEST.COM
Jun 20 19:12:28 dist-test-slave-n3fx krb5kdc[21591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982747, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.126@KRBTEST.COM for HTTP/127.9.155.84@KRBTEST.COM
I20260620 19:12:28.164796 23053 heartbeater.cc:464] Registering TS with master...
I20260620 19:12:28.165120 23053 heartbeater.cc:511] Master 127.9.155.126:41439 requested a full tablet report, sending...
I20260620 19:12:28.166091 22586 ts_manager.cc:194] Registered new tserver with Master: 416cb81100bc4f688bdba356c852a2e8 (127.9.155.67:39833)
I20260620 19:12:28.166900 22586 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.155.67@KRBTEST.COM'} at 127.9.155.67:34755
I20260620 19:12:28.170217 9837 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:12:28.183597 23076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:28.172196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35530 (local address 127.9.155.126:41439)
0620 19:12:28.172519 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:28.172524 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:28.172539 (+ 15us) server_negotiation.cc:407] Connection header received
0620 19:12:28.172591 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:28.172594 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:28.172638 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:28.172709 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:28.174837 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:28.175616 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:28.176919 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:28.177321 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:28.177416 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:28.179289 (+ 1873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:28.179293 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:28.179296 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:28.179330 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:28.181215 (+ 1885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:28.181743 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:28.181747 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:28.181749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:28.181848 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:28.182177 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:28.182180 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:28.182182 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:28.182430 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:28.182561 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:28.183270 (+ 709us) server_negotiation.cc:299] Negotiation successful
0620 19:12:28.183413 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":89,"threads_started":1}
I20260620 19:12:28.188556 22586 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35530:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:12:28.193055 22586 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:12:28.233844 23085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:28.221689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.65:34117 (local address 127.0.0.1:48244)
0620 19:12:28.224980 (+ 3291us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:28.224992 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:28.225066 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:28.226942 (+ 1876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:28.226948 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:28.226972 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:12:28.227116 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:28.227123 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:28.228611 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:28.228615 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:28.231092 (+ 2477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:28.231106 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:28.233222 (+ 2116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:28.233226 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:28.233354 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:28.233397 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 19:12:28.233515 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 19:12:28.233666 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3224,"thread_start_us":102,"threads_started":1}
I20260620 19:12:28.235917 23086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:28.222112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48244 (local address 127.9.155.65:34117)
0620 19:12:28.225307 (+ 3195us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:28.225315 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:28.225338 (+ 23us) server_negotiation.cc:407] Connection header received
0620 19:12:28.225414 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:28.225419 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:28.226554 (+ 1135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:28.226730 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:12:28.227251 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:28.228452 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:28.231705 (+ 3253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:28.232843 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:28.232962 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:28.233142 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:28.235623 (+ 2481us) server_negotiation.cc:299] Negotiation successful
0620 19:12:28.235755 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3083,"thread_start_us":66,"threads_started":1}
I20260620 19:12:28.239528 22719 tablet_service.cc:1467] Processing CreateTablet for tablet 73ec32220f274d36beffb44e281b2113 (DEFAULT_TABLE table=test-table [id=6e0e40cf4d5443d0856eb101d5cfc392]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:12:28.240799 22719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73ec32220f274d36beffb44e281b2113. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:12:28.242743 23083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:28.220414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.66:42347 (local address 127.0.0.1:56404)
0620 19:12:28.224487 (+ 4073us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:28.224594 (+ 107us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:28.224734 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:28.233033 (+ 8299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:28.233037 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:28.233057 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:12:28.233165 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:28.233173 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:28.239406 (+ 6233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:28.239411 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:28.240532 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:28.240546 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:28.242318 (+ 1772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:28.242322 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:28.242417 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:28.242446 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 19:12:28.242496 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 19:12:28.242589 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":1,"thread_start_us":109,"threads_started":1}
I20260620 19:12:28.242914 23087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:28.223707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.67:39833 (local address 127.0.0.1:39456)
0620 19:12:28.231459 (+ 7752us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:28.231476 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:28.231580 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:28.237201 (+ 5621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:28.237206 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:28.237229 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:12:28.237336 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:28.237343 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:28.238870 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:28.238874 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:28.241064 (+ 2190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:28.241075 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:28.242458 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:28.242462 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:28.242558 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:28.242599 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 19:12:28.242654 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 19:12:28.242729 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7638,"mutex_wait_us":851,"thread_start_us":92,"threads_started":1}
I20260620 19:12:28.243628 23088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:28.224690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39456 (local address 127.9.155.67:39833)
0620 19:12:28.236468 (+ 11778us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:28.236476 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:28.236512 (+ 36us) server_negotiation.cc:407] Connection header received
0620 19:12:28.236595 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:28.236601 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:28.236823 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:28.236989 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:12:28.237549 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:28.238718 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:28.241425 (+ 2707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:28.242315 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:28.243172 (+ 857us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:28.243326 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:28.243379 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 19:12:28.243474 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11639,"thread_start_us":83,"threads_started":1}
I20260620 19:12:28.246609 23084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:28.220558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56404 (local address 127.9.155.66:42347)
0620 19:12:28.232452 (+ 11894us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:28.232460 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:28.232489 (+ 29us) server_negotiation.cc:407] Connection header received
0620 19:12:28.232585 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:28.232591 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:28.232842 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:28.233338 (+ 496us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:12:28.233480 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:28.235186 (+ 1706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:28.241250 (+ 6064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:28.242080 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:28.246110 (+ 4030us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:28.246281 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:28.246337 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 19:12:28.246438 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11762,"thread_start_us":118,"threads_started":1}
I20260620 19:12:28.246968 22983 tablet_service.cc:1467] Processing CreateTablet for tablet 73ec32220f274d36beffb44e281b2113 (DEFAULT_TABLE table=test-table [id=6e0e40cf4d5443d0856eb101d5cfc392]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:12:28.248088 22983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73ec32220f274d36beffb44e281b2113. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:12:28.259476 22844 tablet_service.cc:1467] Processing CreateTablet for tablet 73ec32220f274d36beffb44e281b2113 (DEFAULT_TABLE table=test-table [id=6e0e40cf4d5443d0856eb101d5cfc392]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:12:28.260673 22844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73ec32220f274d36beffb44e281b2113. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:12:28.260660 23089 tablet_bootstrap.cc:492] T 73ec32220f274d36beffb44e281b2113 P 2b1c4d9b640440c48b207fd17ca3613e: Bootstrap starting.
I20260620 19:12:28.263082 23089 tablet_bootstrap.cc:654] T 73ec32220f274d36beffb44e281b2113 P 2b1c4d9b640440c48b207fd17ca3613e: Neither blocks nor log segments found. Creating new log.
I20260620 19:12:28.263937 23089 log.cc:824] T 73ec32220f274d36beffb44e281b2113 P 2b1c4d9b640440c48b207fd17ca3613e: Log is configured to *not* fsync() on all Append() calls
I20260620 19:12:28.272709 23090 tablet_bootstrap.cc:492] T 73ec32220f274d36beffb44e281b2113 P 416cb81100bc4f688bdba356c852a2e8: Bootstrap starting.
I20260620 19:12:28.275661 23090 tablet_bootstrap.cc:654] T 73ec32220f274d36beffb44e281b2113 P 416cb81100bc4f688bdba356c852a2e8: Neither blocks nor log segments found. Creating new log.
I20260620 19:12:28.276873 23090 log.cc:824] T 73ec32220f274d36beffb44e281b2113 P 416cb81100bc4f688bdba356c852a2e8: Log is configured to *not* fsync() on all Append() calls
I20260620 19:12:28.282368 23089 tablet_bootstrap.cc:492] T 73ec32220f274d36beffb44e281b2113 P 2b1c4d9b640440c48b207fd17ca3613e: No bootstrap required, opened a new log
I20260620 19:12:28.282622 23089 ts_tablet_manager.cc:1397] T 73ec32220f274d36beffb44e281b2113 P 2b1c4d9b640440c48b207fd17ca3613e: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.000s
I20260620 19:12:28.284631 23092 tablet_bootstrap.cc:492] T 73ec32220f274d36beffb44e281b2113 P 6cf1f2b0ca174b43b623f3189e587d85: Bootstrap starting.
I20260620 19:12:28.286916 23089 raft_consensus.cc:348] T 73ec32220f274d36beffb44e281b2113 P 2b1c4d9b640440c48b207fd17ca3613e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cf1f2b0ca174b43b623f3189e587d85" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 42347 } } peers { permanent_uuid: "2b1c4d9b640440c48b207fd17ca3613e" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 34117 } } peers { permanent_uuid: "416cb81100bc4f688bdba356c852a2e8" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 39833 } }
I20260620 19:12:28.287371 23089 raft_consensus.cc:374] T 73ec32220f274d36beffb44e281b2113 P 2b1c4d9b640440c48b207fd17ca3613e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:12:28.287395 23092 tablet_bootstrap.cc:654] T 73ec32220f274d36beffb44e281b2113 P 6cf1f2b0ca174b43b623f3189e587d85: Neither blocks nor log segments found. Creating new log.
I20260620 19:12:28.287492 23089 raft_consensus.cc:729] T 73ec32220f274d36beffb44e281b2113 P 2b1c4d9b640440c48b207fd17ca3613e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b1c4d9b640440c48b207fd17ca3613e, State: Initialized, Role: FOLLOWER
I20260620 19:12:28.288089 23089 consensus_queue.cc:260] T 73ec32220f274d36beffb44e281b2113 P 2b1c4d9b640440c48b207fd17ca3613e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cf1f2b0ca174b43b623f3189e587d85" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 42347 } } peers { permanent_uuid: "2b1c4d9b640440c48b207fd17ca3613e" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 34117 } } peers { permanent_uuid: "416cb81100bc4f688bdba356c852a2e8" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 39833 } }
I20260620 19:12:28.288268 23092 log.cc:824] T 73ec32220f274d36beffb44e281b2113 P 6cf1f2b0ca174b43b623f3189e587d85: Log is configured to *not* fsync() on all Append() calls
I20260620 19:12:28.289324 23089 ts_tablet_manager.cc:1428] T 73ec32220f274d36beffb44e281b2113 P 2b1c4d9b640440c48b207fd17ca3613e: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260620 19:12:28.289747 22785 heartbeater.cc:503] Master 127.9.155.126:41439 was elected leader, sending a full tablet report...
I20260620 19:12:28.297273 23090 tablet_bootstrap.cc:492] T 73ec32220f274d36beffb44e281b2113 P 416cb81100bc4f688bdba356c852a2e8: No bootstrap required, opened a new log
I20260620 19:12:28.297528 23090 ts_tablet_manager.cc:1397] T 73ec32220f274d36beffb44e281b2113 P 416cb81100bc4f688bdba356c852a2e8: Time spent bootstrapping tablet: real 0.025s user 0.005s sys 0.001s
I20260620 19:12:28.297648 23092 tablet_bootstrap.cc:492] T 73ec32220f274d36beffb44e281b2113 P 6cf1f2b0ca174b43b623f3189e587d85: No bootstrap required, opened a new log
I20260620 19:12:28.297853 23092 ts_tablet_manager.cc:1397] T 73ec32220f274d36beffb44e281b2113 P 6cf1f2b0ca174b43b623f3189e587d85: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.000s
I20260620 19:12:28.301857 23092 raft_consensus.cc:348] T 73ec32220f274d36beffb44e281b2113 P 6cf1f2b0ca174b43b623f3189e587d85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cf1f2b0ca174b43b623f3189e587d85" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 42347 } } peers { permanent_uuid: "2b1c4d9b640440c48b207fd17ca3613e" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 34117 } } peers { permanent_uuid: "416cb81100bc4f688bdba356c852a2e8" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 39833 } }
I20260620 19:12:28.302150 23092 raft_consensus.cc:374] T 73ec32220f274d36beffb44e281b2113 P 6cf1f2b0ca174b43b623f3189e587d85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:12:28.302208 23092 raft_consensus.cc:729] T 73ec32220f274d36beffb44e281b2113 P 6cf1f2b0ca174b43b623f3189e587d85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cf1f2b0ca174b43b623f3189e587d85, State: Initialized, Role: FOLLOWER
I20260620 19:12:28.302197 23090 raft_consensus.cc:348] T 73ec32220f274d36beffb44e281b2113 P 416cb81100bc4f688bdba356c852a2e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cf1f2b0ca174b43b623f3189e587d85" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 42347 } } peers { permanent_uuid: "2b1c4d9b640440c48b207fd17ca3613e" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 34117 } } peers { permanent_uuid: "416cb81100bc4f688bdba356c852a2e8" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 39833 } }
I20260620 19:12:28.302414 23090 raft_consensus.cc:374] T 73ec32220f274d36beffb44e281b2113 P 416cb81100bc4f688bdba356c852a2e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:12:28.302460 23090 raft_consensus.cc:729] T 73ec32220f274d36beffb44e281b2113 P 416cb81100bc4f688bdba356c852a2e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 416cb81100bc4f688bdba356c852a2e8, State: Initialized, Role: FOLLOWER
I20260620 19:12:28.302805 23092 consensus_queue.cc:260] T 73ec32220f274d36beffb44e281b2113 P 6cf1f2b0ca174b43b623f3189e587d85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cf1f2b0ca174b43b623f3189e587d85" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 42347 } } peers { permanent_uuid: "2b1c4d9b640440c48b207fd17ca3613e" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 34117 } } peers { permanent_uuid: "416cb81100bc4f688bdba356c852a2e8" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 39833 } }
I20260620 19:12:28.303802 23090 consensus_queue.cc:260] T 73ec32220f274d36beffb44e281b2113 P 416cb81100bc4f688bdba356c852a2e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cf1f2b0ca174b43b623f3189e587d85" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 42347 } } peers { permanent_uuid: "2b1c4d9b640440c48b207fd17ca3613e" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 34117 } } peers { permanent_uuid: "416cb81100bc4f688bdba356c852a2e8" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 39833 } }
I20260620 19:12:28.303967 23092 ts_tablet_manager.cc:1428] T 73ec32220f274d36beffb44e281b2113 P 6cf1f2b0ca174b43b623f3189e587d85: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260620 19:12:28.304369 22919 heartbeater.cc:503] Master 127.9.155.126:41439 was elected leader, sending a full tablet report...
I20260620 19:12:28.304864 23090 ts_tablet_manager.cc:1428] T 73ec32220f274d36beffb44e281b2113 P 416cb81100bc4f688bdba356c852a2e8: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20260620 19:12:28.305176 23053 heartbeater.cc:503] Master 127.9.155.126:41439 was elected leader, sending a full tablet report...
W20260620 19:12:28.353888 22786 tablet.cc:2369] T 73ec32220f274d36beffb44e281b2113 P 2b1c4d9b640440c48b207fd17ca3613e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 19:12:28.360121 22920 tablet.cc:2369] T 73ec32220f274d36beffb44e281b2113 P 6cf1f2b0ca174b43b623f3189e587d85: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 19:12:28.394377 23054 tablet.cc:2369] T 73ec32220f274d36beffb44e281b2113 P 416cb81100bc4f688bdba356c852a2e8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:12:28.502143 23097 raft_consensus.cc:482] T 73ec32220f274d36beffb44e281b2113 P 416cb81100bc4f688bdba356c852a2e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:12:28.502379 23097 raft_consensus.cc:504] T 73ec32220f274d36beffb44e281b2113 P 416cb81100bc4f688bdba356c852a2e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cf1f2b0ca174b43b623f3189e587d85" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 42347 } } peers { permanent_uuid: "2b1c4d9b640440c48b207fd17ca3613e" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 34117 } } peers { permanent_uuid: "416cb81100bc4f688bdba356c852a2e8" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 39833 } }
I20260620 19:12:28.503834 23097 leader_election.cc:288] T 73ec32220f274d36beffb44e281b2113 P 416cb81100bc4f688bdba356c852a2e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6cf1f2b0ca174b43b623f3189e587d85 (127.9.155.66:42347), 2b1c4d9b640440c48b207fd17ca3613e (127.9.155.65:34117)
I20260620 19:12:28.509470 23084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:28.504259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.67:53643 (local address 127.9.155.66:42347)
0620 19:12:28.504452 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:28.504457 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:28.505001 (+ 544us) server_negotiation.cc:407] Connection header received
0620 19:12:28.505192 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:28.505195 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:28.505243 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:28.505289 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:12:28.505719 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:28.506451 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:28.508189 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:28.508786 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:28.508881 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:28.508933 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:28.509254 (+ 321us) server_negotiation.cc:299] Negotiation successful
0620 19:12:28.509335 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260620 19:12:28.509953 23086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:28.504519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.67:50443 (local address 127.9.155.65:34117)
0620 19:12:28.504631 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:28.504635 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:28.504897 (+ 262us) server_negotiation.cc:407] Connection header received
0620 19:12:28.505134 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:28.505138 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:28.505192 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:28.505240 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:12:28.505707 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:28.506477 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:28.508588 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:28.509241 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:28.509334 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:28.509384 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:28.509731 (+ 347us) server_negotiation.cc:299] Negotiation successful
0620 19:12:28.509820 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 19:12:28.510134 23105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:28.504067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.66:42347 (local address 127.9.155.67:53643)
0620 19:12:28.504863 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:28.504877 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:28.505076 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:28.505418 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:28.505420 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:28.505438 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:12:28.505596 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:28.505602 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:28.506678 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:28.506681 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:28.508052 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:28.508063 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:28.509024 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:28.509027 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:28.509127 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:28.509148 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 19:12:28.509868 (+ 720us) client_negotiation.cc:240] Negotiation successful
0620 19:12:28.509979 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":739,"mutex_wait_us":482,"thread_start_us":118,"threads_started":1}
I20260620 19:12:28.510656 23106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:28.504061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.65:34117 (local address 127.9.155.67:50443)
0620 19:12:28.504801 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:28.504816 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:28.504982 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:28.505387 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:28.505390 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:28.505404 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:12:28.505574 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:28.505582 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:28.506624 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:28.506627 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:28.508445 (+ 1818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:28.508457 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:28.509500 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:28.509504 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:28.509598 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:28.509620 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 19:12:28.510428 (+ 808us) client_negotiation.cc:240] Negotiation successful
0620 19:12:28.510509 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"mutex_wait_us":18,"spinlock_wait_cycles":11008,"thread_start_us":151,"threads_started":1}
I20260620 19:12:28.510799 22873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73ec32220f274d36beffb44e281b2113" candidate_uuid: "416cb81100bc4f688bdba356c852a2e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cf1f2b0ca174b43b623f3189e587d85" is_pre_election: true
I20260620 19:12:28.511152 22873 raft_consensus.cc:2454] T 73ec32220f274d36beffb44e281b2113 P 6cf1f2b0ca174b43b623f3189e587d85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 416cb81100bc4f688bdba356c852a2e8 in term 0.
I20260620 19:12:28.511240 22739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73ec32220f274d36beffb44e281b2113" candidate_uuid: "416cb81100bc4f688bdba356c852a2e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b1c4d9b640440c48b207fd17ca3613e" is_pre_election: true
I20260620 19:12:28.511521 22739 raft_consensus.cc:2454] T 73ec32220f274d36beffb44e281b2113 P 2b1c4d9b640440c48b207fd17ca3613e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 416cb81100bc4f688bdba356c852a2e8 in term 0.
I20260620 19:12:28.511626 22941 leader_election.cc:302] T 73ec32220f274d36beffb44e281b2113 P 416cb81100bc4f688bdba356c852a2e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 416cb81100bc4f688bdba356c852a2e8, 6cf1f2b0ca174b43b623f3189e587d85; no voters:
I20260620 19:12:28.511921 23097 raft_consensus.cc:2781] T 73ec32220f274d36beffb44e281b2113 P 416cb81100bc4f688bdba356c852a2e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:12:28.511991 23097 raft_consensus.cc:482] T 73ec32220f274d36beffb44e281b2113 P 416cb81100bc4f688bdba356c852a2e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:12:28.512027 23097 raft_consensus.cc:3037] T 73ec32220f274d36beffb44e281b2113 P 416cb81100bc4f688bdba356c852a2e8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:12:28.513367 23097 raft_consensus.cc:504] T 73ec32220f274d36beffb44e281b2113 P 416cb81100bc4f688bdba356c852a2e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cf1f2b0ca174b43b623f3189e587d85" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 42347 } } peers { permanent_uuid: "2b1c4d9b640440c48b207fd17ca3613e" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 34117 } } peers { permanent_uuid: "416cb81100bc4f688bdba356c852a2e8" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 39833 } }
I20260620 19:12:28.513759 23097 leader_election.cc:288] T 73ec32220f274d36beffb44e281b2113 P 416cb81100bc4f688bdba356c852a2e8 [CANDIDATE]: Term 1 election: Requested vote from peers 6cf1f2b0ca174b43b623f3189e587d85 (127.9.155.66:42347), 2b1c4d9b640440c48b207fd17ca3613e (127.9.155.65:34117)
I20260620 19:12:28.514222 22873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73ec32220f274d36beffb44e281b2113" candidate_uuid: "416cb81100bc4f688bdba356c852a2e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cf1f2b0ca174b43b623f3189e587d85"
I20260620 19:12:28.514261 22739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73ec32220f274d36beffb44e281b2113" candidate_uuid: "416cb81100bc4f688bdba356c852a2e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b1c4d9b640440c48b207fd17ca3613e"
I20260620 19:12:28.514367 22739 raft_consensus.cc:3037] T 73ec32220f274d36beffb44e281b2113 P 2b1c4d9b640440c48b207fd17ca3613e [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:12:28.514362 22873 raft_consensus.cc:3037] T 73ec32220f274d36beffb44e281b2113 P 6cf1f2b0ca174b43b623f3189e587d85 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:12:28.515666 22739 raft_consensus.cc:2454] T 73ec32220f274d36beffb44e281b2113 P 2b1c4d9b640440c48b207fd17ca3613e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 416cb81100bc4f688bdba356c852a2e8 in term 1.
I20260620 19:12:28.515992 22943 leader_election.cc:302] T 73ec32220f274d36beffb44e281b2113 P 416cb81100bc4f688bdba356c852a2e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b1c4d9b640440c48b207fd17ca3613e, 416cb81100bc4f688bdba356c852a2e8; no voters:
I20260620 19:12:28.516193 23097 raft_consensus.cc:2781] T 73ec32220f274d36beffb44e281b2113 P 416cb81100bc4f688bdba356c852a2e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:12:28.516239 22873 raft_consensus.cc:2454] T 73ec32220f274d36beffb44e281b2113 P 6cf1f2b0ca174b43b623f3189e587d85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 416cb81100bc4f688bdba356c852a2e8 in term 1.
I20260620 19:12:28.516431 23097 raft_consensus.cc:686] T 73ec32220f274d36beffb44e281b2113 P 416cb81100bc4f688bdba356c852a2e8 [term 1 LEADER]: Becoming Leader. State: Replica: 416cb81100bc4f688bdba356c852a2e8, State: Running, Role: LEADER
I20260620 19:12:28.516907 23097 consensus_queue.cc:237] T 73ec32220f274d36beffb44e281b2113 P 416cb81100bc4f688bdba356c852a2e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cf1f2b0ca174b43b623f3189e587d85" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 42347 } } peers { permanent_uuid: "2b1c4d9b640440c48b207fd17ca3613e" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 34117 } } peers { permanent_uuid: "416cb81100bc4f688bdba356c852a2e8" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 39833 } }
I20260620 19:12:28.520859 22586 catalog_manager.cc:5495] T 73ec32220f274d36beffb44e281b2113 P 416cb81100bc4f688bdba356c852a2e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 416cb81100bc4f688bdba356c852a2e8 (127.9.155.67). New cstate: current_term: 1 leader_uuid: "416cb81100bc4f688bdba356c852a2e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cf1f2b0ca174b43b623f3189e587d85" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 42347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b1c4d9b640440c48b207fd17ca3613e" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 34117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "416cb81100bc4f688bdba356c852a2e8" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 39833 } health_report { overall_health: HEALTHY } } }
I20260620 19:12:28.597718 23088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:28.592852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39462 (local address 127.9.155.67:39833)
0620 19:12:28.592975 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:28.592979 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:28.592994 (+ 15us) server_negotiation.cc:407] Connection header received
0620 19:12:28.593184 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:28.593188 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:28.593275 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:28.593401 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:12:28.593660 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:28.594468 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:28.595774 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:28.596109 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:28.596258 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:28.596634 (+ 376us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:12:28.597269 (+ 635us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:12:28.597388 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:28.597456 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 19:12:28.597526 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 19:12:28.610428 22873 raft_consensus.cc:1261] T 73ec32220f274d36beffb44e281b2113 P 6cf1f2b0ca174b43b623f3189e587d85 [term 1 FOLLOWER]: Refusing update from remote peer 416cb81100bc4f688bdba356c852a2e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:12:28.610430 22739 raft_consensus.cc:1261] T 73ec32220f274d36beffb44e281b2113 P 2b1c4d9b640440c48b207fd17ca3613e [term 1 FOLLOWER]: Refusing update from remote peer 416cb81100bc4f688bdba356c852a2e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:12:28.611356 23107 consensus_queue.cc:1035] T 73ec32220f274d36beffb44e281b2113 P 416cb81100bc4f688bdba356c852a2e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b1c4d9b640440c48b207fd17ca3613e" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 34117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:12:28.611636 23097 consensus_queue.cc:1035] T 73ec32220f274d36beffb44e281b2113 P 416cb81100bc4f688bdba356c852a2e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cf1f2b0ca174b43b623f3189e587d85" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 42347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:12:28.626855 23111 mvcc.cc:205] Tried to move back new op lower bound from 7299001338297069568 to 7299001337928339456. Current Snapshot: MvccSnapshot[applied={T|T < 7299001338297069568}]
I20260620 19:12:28.627928 23112 mvcc.cc:205] Tried to move back new op lower bound from 7299001338297069568 to 7299001337928339456. Current Snapshot: MvccSnapshot[applied={T|T < 7299001338297069568}]
I20260620 19:12:28.638284 22587 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35530:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:12:28.638468 22587 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35530:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:12:28.641619 22587 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b7b289d31ad8472aa2f12a3684ec933e: Sending DeleteTablet for 3 replicas of tablet 73ec32220f274d36beffb44e281b2113
I20260620 19:12:28.642729 22844 tablet_service.cc:1514] Processing DeleteTablet for tablet 73ec32220f274d36beffb44e281b2113 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:12:28 UTC) from {username='kudu', principal='kudu/127.9.155.126@KRBTEST.COM'} at 127.0.0.1:56404
I20260620 19:12:28.643000 22719 tablet_service.cc:1514] Processing DeleteTablet for tablet 73ec32220f274d36beffb44e281b2113 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:12:28 UTC) from {username='kudu', principal='kudu/127.9.155.126@KRBTEST.COM'} at 127.0.0.1:48244
I20260620 19:12:28.643198 22983 tablet_service.cc:1514] Processing DeleteTablet for tablet 73ec32220f274d36beffb44e281b2113 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:12:28 UTC) from {username='kudu', principal='kudu/127.9.155.126@KRBTEST.COM'} at 127.0.0.1:39456
I20260620 19:12:28.644704 23122 tablet_replica.cc:330] T 73ec32220f274d36beffb44e281b2113 P 416cb81100bc4f688bdba356c852a2e8: stopping tablet replica
I20260620 19:12:28.644799 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 22625
I20260620 19:12:28.645298 23122 raft_consensus.cc:2229] T 73ec32220f274d36beffb44e281b2113 P 416cb81100bc4f688bdba356c852a2e8 [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:12:28.645859 23122 raft_consensus.cc:2258] T 73ec32220f274d36beffb44e281b2113 P 416cb81100bc4f688bdba356c852a2e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:12:28.647707 23122 ts_tablet_manager.cc:1905] T 73ec32220f274d36beffb44e281b2113 P 416cb81100bc4f688bdba356c852a2e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:12:28.651031 23122 ts_tablet_manager.cc:1918] T 73ec32220f274d36beffb44e281b2113 P 416cb81100bc4f688bdba356c852a2e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:12:28.651230 23122 log.cc:1196] T 73ec32220f274d36beffb44e281b2113 P 416cb81100bc4f688bdba356c852a2e8: Deleting WAL directory at /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-2/wal/wals/73ec32220f274d36beffb44e281b2113
I20260620 19:12:28.651644 23122 ts_tablet_manager.cc:1939] T 73ec32220f274d36beffb44e281b2113 P 416cb81100bc4f688bdba356c852a2e8: Deleting consensus metadata
I20260620 19:12:28.653083 22574 catalog_manager.cc:4841] TS 416cb81100bc4f688bdba356c852a2e8 (127.9.155.67:39833): tablet 73ec32220f274d36beffb44e281b2113 (table test-table [id=6e0e40cf4d5443d0856eb101d5cfc392]) successfully deleted
I20260620 19:12:28.656461 23120 tablet_replica.cc:330] T 73ec32220f274d36beffb44e281b2113 P 6cf1f2b0ca174b43b623f3189e587d85: stopping tablet replica
I20260620 19:12:28.656847 23120 raft_consensus.cc:2229] T 73ec32220f274d36beffb44e281b2113 P 6cf1f2b0ca174b43b623f3189e587d85 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:12:28.657233 23120 raft_consensus.cc:2258] T 73ec32220f274d36beffb44e281b2113 P 6cf1f2b0ca174b43b623f3189e587d85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:12:28.658921 23120 ts_tablet_manager.cc:1905] T 73ec32220f274d36beffb44e281b2113 P 6cf1f2b0ca174b43b623f3189e587d85: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:12:28.661960 23120 ts_tablet_manager.cc:1918] T 73ec32220f274d36beffb44e281b2113 P 6cf1f2b0ca174b43b623f3189e587d85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:12:28.662163 23120 log.cc:1196] T 73ec32220f274d36beffb44e281b2113 P 6cf1f2b0ca174b43b623f3189e587d85: Deleting WAL directory at /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982595231230-9837-0/minicluster-data/ts-1/wal/wals/73ec32220f274d36beffb44e281b2113
I20260620 19:12:28.662528 23120 ts_tablet_manager.cc:1939] T 73ec32220f274d36beffb44e281b2113 P 6cf1f2b0ca174b43b623f3189e587d85: Deleting consensus metadata
I20260620 19:12:28.664012 22572 catalog_manager.cc:4841] TS 6cf1f2b0ca174b43b623f3189e587d85 (127.9.155.66:42347): tablet 73ec32220f274d36beffb44e281b2113 (table test-table [id=6e0e40cf4d5443d0856eb101d5cfc392]) successfully deleted
W20260620 19:12:28.665019 22574 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.155.65:34117 (error 108)
I20260620 19:12:28.666045 23087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:28.665715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.65:34117 (local address 127.0.0.1:48252)
0620 19:12:28.665885 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:28.665946 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.65:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260620 19:12:28.666324 22574 catalog_manager.cc:4584] TS 2b1c4d9b640440c48b207fd17ca3613e (127.9.155.65:34117): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 73ec32220f274d36beffb44e281b2113: Network error: Client connection negotiation failed: client connection to 127.9.155.65:34117: connect: Connection refused (error 111)
I20260620 19:12:28.672457 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 22789
I20260620 19:12:28.679400 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 22922
I20260620 19:12:28.686594 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 22554
I20260620 19:12:28.693023 9837 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 19:12:29.028383 9837 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 19:12:29.028712 9837 mini_ranger.cc:66] Stopping Ranger...
I20260620 19:12:29.363710 9837 mini_ranger.cc:68] Stopped Ranger
2026-06-20 19:12:29.363 UTC [21621] LOG: received smart shutdown request
2026-06-20 19:12:29.367 UTC [21621] LOG: background worker "logical replication launcher" (PID 21630) exited with exit code 1
2026-06-20 19:12:29.367 UTC [21625] LOG: shutting down
2026-06-20 19:12:29.520 UTC [21621] LOG: database system is shut down
2026-06-20T19:12:29Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (47834 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982595231230-9837-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:12:29 dist-test-slave-n3fx krb5kdc[23132](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:12:29 dist-test-slave-n3fx krb5kdc[23132](info): set up 2 sockets
Jun 20 19:12:29 dist-test-slave-n3fx krb5kdc[23132](info): commencing operation
krb5kdc: starting...
W20260620 19:12:30.580315 9837 mini_kdc.cc:121] Time spent starting KDC: real 0.946s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982595231230-9837-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982595231230-9837-0/krb5kdc/test-user.keytab.
Jun 20 19:12:30 dist-test-slave-n3fx krb5kdc[23132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:12:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:12:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.155.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.155.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.155.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab.
Entry for principal kudu/127.9.155.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab.
I20260620 19:12:30.699421 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982595231230-9837-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982595231230-9837-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982595231230-9837-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:45461
--webserver_interface=127.9.155.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab
--principal=kudu/127.9.155.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:45129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.155.126:45461,127.9.155.125:42505,127.9.155.124:45499
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:12:30.810712 23145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:30.811002 23145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:30.811051 23145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:30.815174 23145 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:12:30.815240 23145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:30.815259 23145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:30.815275 23145 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:12:30.815289 23145 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:12:30.820782 23145 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:45129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982595231230-9837-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982595231230-9837-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.155.126:45461,127.9.155.125:42505,127.9.155.124:45499
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab
--principal=kudu/127.9.155.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:45461
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.155.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982595231230-9837-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:12:30.821879 23145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:30.822860 23145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 19:12:30.831558 23145 server_base.cc:1034] running on GCE node
W20260620 19:12:30.831516 23151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:30.831583 23153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:30.831583 23150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:30.832404 23145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:30.833554 23145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:30.834740 23145 hybrid_clock.cc:648] HybridClock initialized: now 1781982750834713 us; error 39 us; skew 500 ppm
Jun 20 19:12:30 dist-test-slave-n3fx krb5kdc[23132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982750, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:30.837520 23145 init.cc:377] Logged in from keytab as kudu/127.9.155.126@KRBTEST.COM (short username kudu)
I20260620 19:12:30.838387 23145 webserver.cc:460] Webserver started at http://127.9.155.126:43869/ using document root <none> and password file <none>
I20260620 19:12:30.838901 23145 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:30.838959 23145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:12:30.839123 23145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:12:30.841162 23145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982595231230-9837-0/minicluster-data/master-0/data/instance:
uuid: "4fcc0a38b4834220ad1fded011f2da88"
format_stamp: "Formatted at 2026-06-20 19:12:30 on dist-test-slave-n3fx"
server_key: "84955b54947b4f2892db337d75cd40e0"
server_key_iv: "434e16aecbb26efb0de4123ddeed7961"
server_key_version: "encryptionkey@0"
I20260620 19:12:30.841670 23145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982595231230-9837-0/minicluster-data/master-0/wal/instance:
uuid: "4fcc0a38b4834220ad1fded011f2da88"
format_stamp: "Formatted at 2026-06-20 19:12:30 on dist-test-slave-n3fx"
server_key: "84955b54947b4f2892db337d75cd40e0"
server_key_iv: "434e16aecbb26efb0de4123ddeed7961"
server_key_version: "encryptionkey@0"
I20260620 19:12:30.845613 23145 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 19:12:30.848279 23160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:30.849548 23145 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:12:30.849663 23145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982595231230-9837-0/minicluster-data/master-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982595231230-9837-0/minicluster-data/master-0/wal
uuid: "4fcc0a38b4834220ad1fded011f2da88"
format_stamp: "Formatted at 2026-06-20 19:12:30 on dist-test-slave-n3fx"
server_key: "84955b54947b4f2892db337d75cd40e0"
server_key_iv: "434e16aecbb26efb0de4123ddeed7961"
server_key_version: "encryptionkey@0"
I20260620 19:12:30.849783 23145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982595231230-9837-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982595231230-9837-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982595231230-9837-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:30.878075 23145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:30.881120 23145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:30.881333 23145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:30.888300 23145 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.126:45461
I20260620 19:12:30.888332 23212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.126:45461 every 8 connection(s)
I20260620 19:12:30.889249 23145 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
I20260620 19:12:30.892056 23213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:12:30.895784 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 23145
I20260620 19:12:30.895900 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982595231230-9837-0/minicluster-data/master-0/wal/instance
I20260620 19:12:30.895833 23213 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.155.126" port: 45461 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:12:30.896148 9837 external_mini_cluster.cc:1428] Setting key aebf717ebe516502b8f119575fe76aca
WARNING: no policy specified for kudu/127.9.155.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.155.125@KRBTEST.COM" created.
Jun 20 19:12:30 dist-test-slave-n3fx krb5kdc[23132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982750, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.126@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:12:30.911145 23216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:30.898259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:45461 (local address 127.0.0.1:57772)
0620 19:12:30.898814 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:30.898861 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:30.899687 (+ 826us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:30.900361 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:30.900366 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:30.900787 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:30.901021 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:30.901034 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:30.902122 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:30.902125 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:30.902827 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:30.902835 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:30.903465 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:30.903469 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:30.903610 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:30.904176 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:12:30.904193 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:30.906382 (+ 2189us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:30.908351 (+ 1969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:30.908360 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:30.908369 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:30.908599 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:30.908947 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:30.908950 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:30.908952 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:30.909066 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:30.909806 (+ 740us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:30.909812 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:30.909942 (+ 130us) client_negotiation.cc:769] Sending connection context
0620 19:12:30.910082 (+ 140us) client_negotiation.cc:240] Negotiation successful
0620 19:12:30.910358 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"mutex_wait_us":3,"thread_start_us":155,"threads_started":1}
I20260620 19:12:30.911149 23217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:30.898259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57772 (local address 127.9.155.126:45461)
0620 19:12:30.898776 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:30.898794 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:30.898953 (+ 159us) server_negotiation.cc:407] Connection header received
0620 19:12:30.899968 (+ 1015us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:30.899972 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:30.900155 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:30.900343 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:30.901152 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:30.902006 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:30.902930 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:30.903327 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:30.903467 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:30.906547 (+ 3080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:30.906553 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:30.906564 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:30.906587 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:30.908245 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:30.908698 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:30.908704 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:30.908706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:30.908837 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:30.909154 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:30.909157 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:30.909159 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:30.909685 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:30.909823 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:30.910197 (+ 374us) server_negotiation.cc:299] Negotiation successful
0620 19:12:30.910433 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":65,"server-negotiator.queue_time_us":405,"spinlock_wait_cycles":30976,"thread_start_us":170,"threads_started":1}
I20260620 19:12:30.913628 23213 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.155.125" port: 42505 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:12:30.914317 23216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:30.914066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.125:42505 (local address 127.0.0.1:57434)
0620 19:12:30.914178 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:30.914253 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.125:42505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 19:12:30.914516 23162 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.155.125:42505: connect: Connection refused (error 111)
I20260620 19:12:30.915498 23216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:30.915269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.125:42505 (local address 127.0.0.1:57438)
0620 19:12:30.915380 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:30.915436 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.125:42505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 19:12:30.915690 23213 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.155.125:42505: Network error: Client connection negotiation failed: client connection to 127.9.155.125:42505: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.155.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.125.keytab.
Entry for principal kudu/127.9.155.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.125.keytab.
I20260620 19:12:30.920951 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982595231230-9837-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982595231230-9837-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982595231230-9837-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982595231230-9837-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.125:42505
--webserver_interface=127.9.155.125
--webserver_port=0
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.125.keytab
--principal=kudu/127.9.155.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:45129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.155.126:45461,127.9.155.125:42505,127.9.155.124:45499
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:12:30.964929 23213 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.155.125" port: 42505 } attempt: 1
I20260620 19:12:30.965853 23216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:30.965519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.125:42505 (local address 127.0.0.1:57450)
0620 19:12:30.965674 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:30.965773 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.125:42505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 19:12:30.966605 23216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:30.966372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.125:42505 (local address 127.0.0.1:57454)
0620 19:12:30.966496 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:30.966542 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.125:42505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 19:12:30.966723 23213 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.155.125:42505: Network error: Client connection negotiation failed: client connection to 127.9.155.125:42505: connect: Connection refused (error 111)
W20260620 19:12:31.030884 23220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:31.031153 23220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:31.031212 23220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 19:12:31.034862 23213 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.155.125" port: 42505 } attempt: 2
W20260620 19:12:31.035360 23220 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:12:31.035430 23220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:31.035457 23220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:31.035533 23220 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:12:31.035567 23220 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:12:31.035751 23216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:31.035402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.125:42505 (local address 127.0.0.1:57456)
0620 19:12:31.035572 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:31.035647 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.125:42505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 19:12:31.036414 23216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:31.036247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.125:42505 (local address 127.0.0.1:57466)
0620 19:12:31.036321 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:31.036363 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.125:42505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260620 19:12:31.036521 23213 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.155.125:42505: Network error: Client connection negotiation failed: client connection to 127.9.155.125:42505: connect: Connection refused (error 111)
I20260620 19:12:31.041358 23220 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:45129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982595231230-9837-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982595231230-9837-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.9.155.126:45461,127.9.155.125:42505,127.9.155.124:45499
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.125.keytab
--principal=kudu/127.9.155.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.125:42505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982595231230-9837-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.9.155.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23220
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982595231230-9837-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:12:31.042536 23220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:31.043735 23220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:12:31.052174 23225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:31.052184 23226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:31.052248 23228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:31.052443 23220 server_base.cc:1034] running on GCE node
I20260620 19:12:31.053136 23220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:31.054297 23220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:31.055460 23220 hybrid_clock.cc:648] HybridClock initialized: now 1781982751055443 us; error 33 us; skew 500 ppm
Jun 20 19:12:31 dist-test-slave-n3fx krb5kdc[23132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982751, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:31.058370 23220 init.cc:377] Logged in from keytab as kudu/127.9.155.125@KRBTEST.COM (short username kudu)
I20260620 19:12:31.059360 23220 webserver.cc:460] Webserver started at http://127.9.155.125:42013/ using document root <none> and password file <none>
I20260620 19:12:31.059908 23220 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:31.059975 23220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:12:31.060170 23220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:12:31.062223 23220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982595231230-9837-0/minicluster-data/master-1/data/instance:
uuid: "46103ef2a93e4a3e808e01f82d3273d6"
format_stamp: "Formatted at 2026-06-20 19:12:31 on dist-test-slave-n3fx"
server_key: "be934c4fab45f583eb027d577449392f"
server_key_iv: "20c9a9330f8c712e77503c59d6e4d457"
server_key_version: "encryptionkey@0"
I20260620 19:12:31.062750 23220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982595231230-9837-0/minicluster-data/master-1/wal/instance:
uuid: "46103ef2a93e4a3e808e01f82d3273d6"
format_stamp: "Formatted at 2026-06-20 19:12:31 on dist-test-slave-n3fx"
server_key: "be934c4fab45f583eb027d577449392f"
server_key_iv: "20c9a9330f8c712e77503c59d6e4d457"
server_key_version: "encryptionkey@0"
I20260620 19:12:31.066562 23220 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 19:12:31.069187 23235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:31.070814 23220 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:12:31.070945 23220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982595231230-9837-0/minicluster-data/master-1/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982595231230-9837-0/minicluster-data/master-1/wal
uuid: "46103ef2a93e4a3e808e01f82d3273d6"
format_stamp: "Formatted at 2026-06-20 19:12:31 on dist-test-slave-n3fx"
server_key: "be934c4fab45f583eb027d577449392f"
server_key_iv: "20c9a9330f8c712e77503c59d6e4d457"
server_key_version: "encryptionkey@0"
I20260620 19:12:31.071054 23220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982595231230-9837-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982595231230-9837-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982595231230-9837-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:31.082338 23220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:31.085451 23220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:31.085635 23220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:31.092767 23220 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.125:42505
I20260620 19:12:31.092784 23287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.125:42505 every 8 connection(s)
I20260620 19:12:31.093760 23220 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982595231230-9837-0/minicluster-data/master-1/data/info.pb
I20260620 19:12:31.096544 23288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:12:31.096817 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 23220
I20260620 19:12:31.096938 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982595231230-9837-0/minicluster-data/master-1/wal/instance
I20260620 19:12:31.097200 9837 external_mini_cluster.cc:1428] Setting key 94b96665816fdfa9c128577d5e631305
I20260620 19:12:31.101138 23288 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.155.126" port: 45461 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.9.155.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.155.124@KRBTEST.COM" created.
Jun 20 19:12:31 dist-test-slave-n3fx krb5kdc[23132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982751, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.125@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:12:31.117082 23217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:31.103528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57774 (local address 127.9.155.126:45461)
0620 19:12:31.103691 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:31.103695 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:31.104545 (+ 850us) server_negotiation.cc:407] Connection header received
0620 19:12:31.105579 (+ 1034us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:31.105582 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:31.105692 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:31.105785 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:31.107088 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:31.107674 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:31.109070 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:31.109363 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:31.109469 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:31.112561 (+ 3092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:31.112565 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:31.112567 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:31.112600 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:31.114751 (+ 2151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:31.115322 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:31.115335 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:31.115337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:31.115438 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:31.115781 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:31.115784 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:31.115786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:31.116025 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:31.116128 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:31.116760 (+ 632us) server_negotiation.cc:299] Negotiation successful
0620 19:12:31.116891 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 19:12:31.117796 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:31.103837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:45461 (local address 127.0.0.1:57774)
0620 19:12:31.104368 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:31.104420 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:31.105355 (+ 935us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:31.105977 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:31.105985 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:31.106359 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:31.106913 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:31.106929 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:31.107842 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:31.107845 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:31.108944 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:31.108956 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:31.109534 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:31.109538 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:31.109769 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:31.110437 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:12:31.110460 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:31.112364 (+ 1904us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:31.114885 (+ 2521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:31.114891 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:31.114906 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:31.115205 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:31.115532 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:31.115535 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:31.115536 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:31.115682 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:31.116115 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:31.116121 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:31.116484 (+ 363us) client_negotiation.cc:769] Sending connection context
0620 19:12:31.116648 (+ 164us) client_negotiation.cc:240] Negotiation successful
0620 19:12:31.116922 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":114,"threads_started":1}
I20260620 19:12:31.119153 23288 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.155.125" port: 42505 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 19:12:31 dist-test-slave-n3fx krb5kdc[23132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982751, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.125@KRBTEST.COM for kudu/127.9.155.125@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.155.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.124.keytab.
Entry for principal kudu/127.9.155.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.124.keytab.
I20260620 19:12:31.126722 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982595231230-9837-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982595231230-9837-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982595231230-9837-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982595231230-9837-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.124:45499
--webserver_interface=127.9.155.124
--webserver_port=0
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.124.keytab
--principal=kudu/127.9.155.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:45129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.155.126:45461,127.9.155.125:42505,127.9.155.124:45499
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:12:31.127667 23213 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.155.125" port: 42505 } attempt: 3
I20260620 19:12:31.128342 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:31.119694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.125:42505 (local address 127.0.0.1:57472)
0620 19:12:31.119840 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:31.119854 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:31.119932 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:31.120522 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:31.120524 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:31.120655 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:31.120747 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:31.120752 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:31.121831 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:31.121834 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:31.122404 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:31.122412 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:31.123069 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:31.123071 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:31.123139 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:31.123620 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:12:31.123631 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:31.124964 (+ 1333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:31.126778 (+ 1814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:31.126781 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:31.126783 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:31.127006 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:31.127264 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:31.127267 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:31.127270 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:31.127342 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:31.127836 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:31.127839 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:31.127927 (+ 88us) client_negotiation.cc:769] Sending connection context
0620 19:12:31.128023 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 19:12:31.128153 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260620 19:12:31.128546 23293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:31.119807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57472 (local address 127.9.155.125:42505)
0620 19:12:31.120115 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:31.120124 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:31.120143 (+ 19us) server_negotiation.cc:407] Connection header received
0620 19:12:31.120176 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:31.120180 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:31.120380 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:31.120522 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:31.120831 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:31.121722 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:31.122487 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:31.122957 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:31.123055 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:31.125086 (+ 2031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:31.125095 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:31.125108 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:31.125127 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:31.126675 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:31.127108 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:31.127113 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:31.127114 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:31.127185 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:31.127413 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:31.127416 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:31.127417 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:31.127668 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:31.127800 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:31.128147 (+ 347us) server_negotiation.cc:299] Negotiation successful
0620 19:12:31.128331 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":113,"threads_started":1}
I20260620 19:12:31.130765 23288 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.155.124" port: 45499 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:12:31.131503 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:31.131217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.124:45499 (local address 127.0.0.1:36464)
0620 19:12:31.131336 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:31.131422 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.124:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 19:12:31.131680 23237 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.155.124:45499: connect: Connection refused (error 111)
I20260620 19:12:31.132630 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:31.132409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.124:45499 (local address 127.0.0.1:36466)
0620 19:12:31.132517 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:31.132568 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.124:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
Jun 20 19:12:31 dist-test-slave-n3fx krb5kdc[23132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982750, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.126@KRBTEST.COM for kudu/127.9.155.125@KRBTEST.COM
W20260620 19:12:31.133400 23288 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.155.124:45499: Network error: Client connection negotiation failed: client connection to 127.9.155.124:45499: connect: Connection refused (error 111)
I20260620 19:12:31.137410 23293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:31.128709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57488 (local address 127.9.155.125:42505)
0620 19:12:31.128995 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:31.128998 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:31.129011 (+ 13us) server_negotiation.cc:407] Connection header received
0620 19:12:31.129044 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:31.129046 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:31.129086 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:31.129158 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:31.129566 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:31.130062 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:31.130939 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:31.131236 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:31.131688 (+ 452us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:31.133845 (+ 2157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:31.133848 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:31.133850 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:31.133871 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:31.135750 (+ 1879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:31.136272 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:31.136276 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:31.136278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:31.136361 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:31.136654 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:31.136656 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:31.136658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:31.136843 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:31.136923 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:31.137202 (+ 279us) server_negotiation.cc:299] Negotiation successful
0620 19:12:31.137297 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":167,"threads_started":1}
I20260620 19:12:31.137486 23216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:31.128158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.125:42505 (local address 127.0.0.1:57488)
0620 19:12:31.128458 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:31.128472 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:31.128556 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:31.129205 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:31.129208 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:31.129357 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:31.129461 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:31.129467 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:31.130161 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:31.130165 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:31.130816 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:31.130827 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:31.131364 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:31.131368 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:31.131459 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:31.131986 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:12:31.132002 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:31.133720 (+ 1718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:31.135859 (+ 2139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:31.135863 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:31.135866 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:31.136129 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:31.136450 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:31.136454 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:31.136456 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:31.136569 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:31.136927 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:31.136930 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:31.137046 (+ 116us) client_negotiation.cc:769] Sending connection context
0620 19:12:31.137166 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 19:12:31.137320 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217}
I20260620 19:12:31.138259 23213 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.155.124" port: 45499 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:12:31.138935 23216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:31.138714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.124:45499 (local address 127.0.0.1:36470)
0620 19:12:31.138826 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:31.138875 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.124:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 19:12:31.139549 23216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:31.139337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.124:45499 (local address 127.0.0.1:36484)
0620 19:12:31.139463 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:31.139502 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.124:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260620 19:12:31.139673 23213 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.155.124:45499: Network error: Client connection negotiation failed: client connection to 127.9.155.124:45499: connect: Connection refused (error 111)
I20260620 19:12:31.170884 23213 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.155.124" port: 45499 } attempt: 1
I20260620 19:12:31.171756 23216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:31.171465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.124:45499 (local address 127.0.0.1:36488)
0620 19:12:31.171604 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:31.171666 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.124:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 19:12:31.172442 23216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:31.172249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.124:45499 (local address 127.0.0.1:36496)
0620 19:12:31.172338 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:31.172383 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.124:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260620 19:12:31.172580 23213 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.155.124:45499: Network error: Client connection negotiation failed: client connection to 127.9.155.124:45499: connect: Connection refused (error 111)
I20260620 19:12:31.182642 23288 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.155.124" port: 45499 } attempt: 1
I20260620 19:12:31.183405 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:31.183132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.124:45499 (local address 127.0.0.1:36512)
0620 19:12:31.183266 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:31.183331 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.124:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 19:12:31.184181 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:31.183982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.124:45499 (local address 127.0.0.1:36526)
0620 19:12:31.184073 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:31.184119 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.124:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260620 19:12:31.184316 23288 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.155.124:45499: Network error: Client connection negotiation failed: client connection to 127.9.155.124:45499: connect: Connection refused (error 111)
W20260620 19:12:31.240437 23294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:31.240708 23294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:31.240749 23294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:31.244835 23294 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:12:31.244902 23294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:31.244921 23294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:31.244937 23294 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:12:31.244956 23294 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:12:31.247785 23213 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.155.124" port: 45499 } attempt: 2
I20260620 19:12:31.248803 23216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:31.248411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.124:45499 (local address 127.0.0.1:36540)
0620 19:12:31.248568 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:31.248635 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.124:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 19:12:31.249475 23216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:31.249280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.124:45499 (local address 127.0.0.1:36546)
0620 19:12:31.249373 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:31.249422 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.124:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260620 19:12:31.249580 23213 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.155.124:45499: Network error: Client connection negotiation failed: client connection to 127.9.155.124:45499: connect: Connection refused (error 111)
I20260620 19:12:31.250811 23294 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:45129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982595231230-9837-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982595231230-9837-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.9.155.126:45461,127.9.155.125:42505,127.9.155.124:45499
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.124.keytab
--principal=kudu/127.9.155.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.124:45499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982595231230-9837-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.155.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23294
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982595231230-9837-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:12:31.251818 23294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:31.252450 23288 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.155.124" port: 45499 } attempt: 2
I20260620 19:12:31.252842 23294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 19:12:31.253123 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:31.252865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.124:45499 (local address 127.0.0.1:36556)
0620 19:12:31.252998 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:31.253049 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.124:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 19:12:31.253732 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:31.253539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.124:45499 (local address 127.0.0.1:36568)
0620 19:12:31.253629 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:31.253673 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.124:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260620 19:12:31.253878 23288 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.155.124:45499: Network error: Client connection negotiation failed: client connection to 127.9.155.124:45499: connect: Connection refused (error 111)
W20260620 19:12:31.261204 23302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:31.261231 23301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:31.261374 23294 server_base.cc:1034] running on GCE node
W20260620 19:12:31.261324 23304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:31.262045 23294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:31.263236 23294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:31.264386 23294 hybrid_clock.cc:648] HybridClock initialized: now 1781982751264366 us; error 41 us; skew 500 ppm
Jun 20 19:12:31 dist-test-slave-n3fx krb5kdc[23132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982751, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:31.267203 23294 init.cc:377] Logged in from keytab as kudu/127.9.155.124@KRBTEST.COM (short username kudu)
I20260620 19:12:31.268076 23294 webserver.cc:460] Webserver started at http://127.9.155.124:43487/ using document root <none> and password file <none>
I20260620 19:12:31.268620 23294 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:31.268720 23294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:12:31.268930 23294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:12:31.270951 23294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982595231230-9837-0/minicluster-data/master-2/data/instance:
uuid: "e90aa5a6b5f14ea09f9f4ac55f55a7b5"
format_stamp: "Formatted at 2026-06-20 19:12:31 on dist-test-slave-n3fx"
server_key: "86edabce61f3768c5cd5fdc658fb3fe0"
server_key_iv: "4d17cfbb0b76be33073ed07abdaf51b1"
server_key_version: "encryptionkey@0"
I20260620 19:12:31.271471 23294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982595231230-9837-0/minicluster-data/master-2/wal/instance:
uuid: "e90aa5a6b5f14ea09f9f4ac55f55a7b5"
format_stamp: "Formatted at 2026-06-20 19:12:31 on dist-test-slave-n3fx"
server_key: "86edabce61f3768c5cd5fdc658fb3fe0"
server_key_iv: "4d17cfbb0b76be33073ed07abdaf51b1"
server_key_version: "encryptionkey@0"
I20260620 19:12:31.275290 23294 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260620 19:12:31.277705 23311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:31.278884 23294 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 19:12:31.278981 23294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982595231230-9837-0/minicluster-data/master-2/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982595231230-9837-0/minicluster-data/master-2/wal
uuid: "e90aa5a6b5f14ea09f9f4ac55f55a7b5"
format_stamp: "Formatted at 2026-06-20 19:12:31 on dist-test-slave-n3fx"
server_key: "86edabce61f3768c5cd5fdc658fb3fe0"
server_key_iv: "4d17cfbb0b76be33073ed07abdaf51b1"
server_key_version: "encryptionkey@0"
I20260620 19:12:31.279099 23294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982595231230-9837-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982595231230-9837-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982595231230-9837-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:31.309505 23294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:31.312534 23294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:31.312707 23294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:31.319828 23294 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.124:45499
I20260620 19:12:31.319903 23363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.124:45499 every 8 connection(s)
I20260620 19:12:31.320837 23294 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982595231230-9837-0/minicluster-data/master-2/data/info.pb
I20260620 19:12:31.323742 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 23294
I20260620 19:12:31.323787 23364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:12:31.323860 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982595231230-9837-0/minicluster-data/master-2/wal/instance
I20260620 19:12:31.324153 9837 external_mini_cluster.cc:1428] Setting key acc781e44bd95ca676ffd7ec72d115ca
I20260620 19:12:31.328960 23364 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.155.126" port: 45461 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 19:12:31 dist-test-slave-n3fx krb5kdc[23132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.155.124@KRBTEST.COM
Jun 20 19:12:31 dist-test-slave-n3fx krb5kdc[23132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982751, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.124@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:12:31.339836 23367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:31.325830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36570 (local address 127.9.155.124:45499)
0620 19:12:31.326312 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:31.326331 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:31.326378 (+ 47us) server_negotiation.cc:407] Connection header received
0620 19:12:31.327189 (+ 811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:31.327217 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:31.327702 (+ 485us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:31.328099 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:31.328879 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:31.330319 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:31.331457 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:31.332134 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:31.332315 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:31.335047 (+ 2732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:31.335052 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:31.335061 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:31.335087 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:31.336814 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:31.337274 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:31.337279 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:31.337284 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:31.337387 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:31.337641 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:31.337644 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:31.337646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:31.338139 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:31.338359 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:31.338574 (+ 215us) server_negotiation.cc:299] Negotiation successful
0620 19:12:31.338862 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":139,"threads_started":1}
I20260620 19:12:31.340762 23368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:31.331093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:45461 (local address 127.0.0.1:57776)
0620 19:12:31.331554 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:31.331601 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:31.331787 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:31.332054 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:31.332059 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:31.332519 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:31.332683 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:31.332690 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:31.333719 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:31.333722 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:31.334648 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:31.334658 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:31.335155 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:31.335158 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:31.335267 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:31.335764 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:12:31.335778 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:31.337406 (+ 1628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:31.339221 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:31.339228 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:31.339233 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:31.339433 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:31.339727 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:31.339729 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:31.339730 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:31.339796 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:31.340202 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:31.340208 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:31.340352 (+ 144us) client_negotiation.cc:769] Sending connection context
0620 19:12:31.340463 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 19:12:31.340594 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":197,"threads_started":1}
I20260620 19:12:31.340802 23217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:31.331139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57776 (local address 127.9.155.126:45461)
0620 19:12:31.331261 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:31.331265 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:31.331759 (+ 494us) server_negotiation.cc:407] Connection header received
0620 19:12:31.331875 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:31.331877 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:31.331924 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:31.331989 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:31.332810 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:31.333568 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:31.334757 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:31.335063 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:31.335137 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:31.337540 (+ 2403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:31.337542 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:31.337544 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:31.337568 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:31.339115 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:31.339519 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:31.339523 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:31.339524 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:31.339618 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:31.339875 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:31.339878 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:31.339879 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:31.340104 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:31.340200 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:31.340536 (+ 336us) server_negotiation.cc:299] Negotiation successful
0620 19:12:31.340659 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 19:12:31.342021 23364 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.155.125" port: 42505 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:12:31.345048 23288 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.155.124" port: 45499 } attempt: 3
Jun 20 19:12:31 dist-test-slave-n3fx krb5kdc[23132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982751, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.124@KRBTEST.COM for kudu/127.9.155.125@KRBTEST.COM
Jun 20 19:12:31 dist-test-slave-n3fx krb5kdc[23132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:12:31.348944 23213 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.155.124" port: 45499 } attempt: 3
Jun 20 19:12:31 dist-test-slave-n3fx krb5kdc[23132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982751, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.125@KRBTEST.COM for kudu/127.9.155.124@KRBTEST.COM
I20260620 19:12:31.351410 23368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:31.342520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.125:42505 (local address 127.0.0.1:57500)
0620 19:12:31.342622 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:31.342635 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:31.342703 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:31.343022 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:31.343025 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:31.343179 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:31.343614 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:31.343623 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:31.344424 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:31.344427 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:31.344924 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:31.344932 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:31.345594 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:31.345598 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:31.345705 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:31.346157 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:12:31.346171 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:31.348032 (+ 1861us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:31.349822 (+ 1790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:31.349825 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:31.349827 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:31.350050 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:31.350365 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:31.350368 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:31.350370 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:31.350458 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:31.350922 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:31.350925 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:31.351006 (+ 81us) client_negotiation.cc:769] Sending connection context
0620 19:12:31.351090 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 19:12:31.351215 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 19:12:31.351410 23293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:31.342702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57500 (local address 127.9.155.125:42505)
0620 19:12:31.342832 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:31.342836 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:31.342848 (+ 12us) server_negotiation.cc:407] Connection header received
0620 19:12:31.342883 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:31.342885 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:31.342921 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:31.342994 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:31.343741 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:31.344309 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:31.345053 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:31.345385 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:31.345475 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:31.348169 (+ 2694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:31.348173 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:31.348175 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:31.348202 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:31.349704 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:31.350161 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:31.350165 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:31.350166 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:31.350250 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:31.350556 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:31.350559 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:31.350561 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:31.350786 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:31.350851 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:31.351166 (+ 315us) server_negotiation.cc:299] Negotiation successful
0620 19:12:31.351266 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 19:12:31.352563 23364 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.155.124" port: 45499 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:12:31.353111 23217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:31.342317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57786 (local address 127.9.155.126:45461)
0620 19:12:31.342828 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:31.342832 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:31.342841 (+ 9us) server_negotiation.cc:407] Connection header received
0620 19:12:31.342867 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:31.342869 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:31.342903 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:31.342960 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:31.343572 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:31.344119 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:31.345132 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:31.345543 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:31.345640 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:31.348237 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:31.348241 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:31.348244 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:31.348269 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:31.349930 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:31.350422 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:31.350425 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:31.350427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:31.350501 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:31.350792 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:31.350795 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:31.350797 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:31.351211 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:31.351423 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:31.352827 (+ 1404us) server_negotiation.cc:299] Negotiation successful
0620 19:12:31.352949 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":176,"server-negotiator.queue_time_us":447}
I20260620 19:12:31.354743 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:31.345483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.124:45499 (local address 127.0.0.1:36576)
0620 19:12:31.345701 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:31.345714 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:31.345796 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:31.346344 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:31.346347 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:31.346523 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:31.346627 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:31.346634 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:31.347545 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:31.347549 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:31.348326 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:31.348336 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:31.348834 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:31.348838 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:31.348921 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:31.349770 (+ 849us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:12:31.349784 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:31.351094 (+ 1310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:31.352925 (+ 1831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:31.352928 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:31.352930 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:31.353165 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:31.353496 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:31.353500 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:31.353502 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:31.353588 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:31.354296 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:31.354299 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:31.354388 (+ 89us) client_negotiation.cc:769] Sending connection context
0620 19:12:31.354464 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 19:12:31.354590 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150}
I20260620 19:12:31.355511 23367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:31.345594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36576 (local address 127.9.155.124:45499)
0620 19:12:31.345715 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:31.345719 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:31.345757 (+ 38us) server_negotiation.cc:407] Connection header received
0620 19:12:31.345880 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:31.345883 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:31.345930 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:31.345982 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:31.346763 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:31.347443 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:31.348439 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:31.348706 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:31.348779 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:31.351273 (+ 2494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:31.351277 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:31.351279 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:31.351301 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:31.352821 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:31.353285 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:31.353288 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:31.353290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:31.353392 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:31.353696 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:31.353699 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:31.353700 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:31.354189 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:31.355199 (+ 1010us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:31.355266 (+ 67us) server_negotiation.cc:299] Negotiation successful
0620 19:12:31.355374 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
Jun 20 19:12:31 dist-test-slave-n3fx krb5kdc[23132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982750, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.126@KRBTEST.COM for kudu/127.9.155.124@KRBTEST.COM
I20260620 19:12:31.359241 23288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46103ef2a93e4a3e808e01f82d3273d6: Bootstrap starting.
I20260620 19:12:31.360307 23216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:31.349307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.124:45499 (local address 127.0.0.1:36580)
0620 19:12:31.349483 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:31.349496 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:31.349565 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:31.350657 (+ 1092us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:31.350660 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:31.350816 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:31.350982 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:31.350991 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:31.353673 (+ 2682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:31.353677 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:31.354414 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:31.354424 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:31.355010 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:31.355013 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:31.355096 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:31.355529 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:12:31.355543 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:31.357200 (+ 1657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:31.358860 (+ 1660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:31.358863 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:31.358865 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:31.359130 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:31.359382 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:31.359385 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:31.359387 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:31.359464 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:31.359797 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:31.359801 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:31.359889 (+ 88us) client_negotiation.cc:769] Sending connection context
0620 19:12:31.359970 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 19:12:31.360107 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"mutex_wait_us":3}
I20260620 19:12:31.360319 23369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:31.349780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36580 (local address 127.9.155.124:45499)
0620 19:12:31.350408 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:31.350417 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:31.350429 (+ 12us) server_negotiation.cc:407] Connection header received
0620 19:12:31.350461 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:31.350463 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:31.350503 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:31.351479 (+ 976us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:31.351566 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:31.352353 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:31.354536 (+ 2183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:31.354901 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:31.354995 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:31.357324 (+ 2329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:31.357327 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:31.357329 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:31.357349 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:31.358762 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:31.359219 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:31.359223 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:31.359225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:31.359304 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:31.359540 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:31.359544 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:31.359545 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:31.359713 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:31.359794 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:31.360029 (+ 235us) server_negotiation.cc:299] Negotiation successful
0620 19:12:31.360131 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":101,"threads_started":1}
Jun 20 19:12:31 dist-test-slave-n3fx krb5kdc[23132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.155.125@KRBTEST.COM
I20260620 19:12:31.362118 23288 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46103ef2a93e4a3e808e01f82d3273d6: Neither blocks nor log segments found. Creating new log.
I20260620 19:12:31.362941 23288 log.cc:824] T 00000000000000000000000000000000 P 46103ef2a93e4a3e808e01f82d3273d6: Log is configured to *not* fsync() on all Append() calls
I20260620 19:12:31.363224 23213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fcc0a38b4834220ad1fded011f2da88: Bootstrap starting.
I20260620 19:12:31.364616 23293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:31.354834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57508 (local address 127.9.155.125:42505)
0620 19:12:31.354921 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:31.354924 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:31.355191 (+ 267us) server_negotiation.cc:407] Connection header received
0620 19:12:31.355346 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:31.355349 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:31.355389 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:31.355434 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:31.356166 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:31.356855 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:31.358226 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:31.358553 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:31.358661 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:31.361152 (+ 2491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:31.361155 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:31.361157 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:31.361182 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:31.362635 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:31.363066 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:31.363069 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:31.363071 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:31.363163 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:31.363555 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:31.363561 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:31.363563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:31.363776 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:31.363935 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:31.364353 (+ 418us) server_negotiation.cc:299] Negotiation successful
0620 19:12:31.364472 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
Jun 20 19:12:31 dist-test-slave-n3fx krb5kdc[23132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982751, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.124@KRBTEST.COM for kudu/127.9.155.124@KRBTEST.COM
I20260620 19:12:31.365774 23213 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4fcc0a38b4834220ad1fded011f2da88: Neither blocks nor log segments found. Creating new log.
I20260620 19:12:31.366397 23288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46103ef2a93e4a3e808e01f82d3273d6: No bootstrap required, opened a new log
I20260620 19:12:31.366555 23213 log.cc:824] T 00000000000000000000000000000000 P 4fcc0a38b4834220ad1fded011f2da88: Log is configured to *not* fsync() on all Append() calls
I20260620 19:12:31.368022 9837 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:12:31.368728 23213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fcc0a38b4834220ad1fded011f2da88: No bootstrap required, opened a new log
I20260620 19:12:31.370364 23288 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46103ef2a93e4a3e808e01f82d3273d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fcc0a38b4834220ad1fded011f2da88" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 45461 } } peers { permanent_uuid: "46103ef2a93e4a3e808e01f82d3273d6" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 42505 } } peers { permanent_uuid: "e90aa5a6b5f14ea09f9f4ac55f55a7b5" member_type: VOTER last_known_addr { host: "127.9.155.124" port: 45499 } }
I20260620 19:12:31.370600 23288 raft_consensus.cc:374] T 00000000000000000000000000000000 P 46103ef2a93e4a3e808e01f82d3273d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:12:31.370682 23288 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46103ef2a93e4a3e808e01f82d3273d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46103ef2a93e4a3e808e01f82d3273d6, State: Initialized, Role: FOLLOWER
I20260620 19:12:31.371184 23288 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46103ef2a93e4a3e808e01f82d3273d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fcc0a38b4834220ad1fded011f2da88" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 45461 } } peers { permanent_uuid: "46103ef2a93e4a3e808e01f82d3273d6" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 42505 } } peers { permanent_uuid: "e90aa5a6b5f14ea09f9f4ac55f55a7b5" member_type: VOTER last_known_addr { host: "127.9.155.124" port: 45499 } }
I20260620 19:12:31.372430 23213 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4fcc0a38b4834220ad1fded011f2da88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fcc0a38b4834220ad1fded011f2da88" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 45461 } } peers { permanent_uuid: "46103ef2a93e4a3e808e01f82d3273d6" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 42505 } } peers { permanent_uuid: "e90aa5a6b5f14ea09f9f4ac55f55a7b5" member_type: VOTER last_known_addr { host: "127.9.155.124" port: 45499 } }
I20260620 19:12:31.372742 23213 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4fcc0a38b4834220ad1fded011f2da88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:12:31.372830 23213 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4fcc0a38b4834220ad1fded011f2da88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fcc0a38b4834220ad1fded011f2da88, State: Initialized, Role: FOLLOWER
I20260620 19:12:31.373376 23213 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4fcc0a38b4834220ad1fded011f2da88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fcc0a38b4834220ad1fded011f2da88" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 45461 } } peers { permanent_uuid: "46103ef2a93e4a3e808e01f82d3273d6" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 42505 } } peers { permanent_uuid: "e90aa5a6b5f14ea09f9f4ac55f55a7b5" member_type: VOTER last_known_addr { host: "127.9.155.124" port: 45499 } }
I20260620 19:12:31.374878 23213 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4fcc0a38b4834220ad1fded011f2da88 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:12:31.375905 23368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:31.353004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.124:45499 (local address 127.0.0.1:36592)
0620 19:12:31.353345 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:31.353360 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:31.353428 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:31.353835 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:31.353838 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:31.353967 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:31.354035 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:31.354039 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:31.355010 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:31.355013 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:31.355760 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:31.355771 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:31.356490 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:31.356493 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:31.356564 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:31.363571 (+ 7007us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:12:31.363585 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:31.365012 (+ 1427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:31.374481 (+ 9469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:31.374485 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:31.374487 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:31.374727 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:31.374994 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:31.374997 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:31.375000 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:31.375078 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:31.375455 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:31.375458 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:31.375523 (+ 65us) client_negotiation.cc:769] Sending connection context
0620 19:12:31.375589 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 19:12:31.375732 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102}
I20260620 19:12:31.377089 23370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:31.353051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36592 (local address 127.9.155.124:45499)
0620 19:12:31.353646 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:31.353655 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:31.353667 (+ 12us) server_negotiation.cc:407] Connection header received
0620 19:12:31.353696 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:31.353698 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:31.353734 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:31.354114 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:31.354230 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:31.354912 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:31.356054 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:31.356385 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:31.372344 (+ 15959us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:31.372476 (+ 132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:31.372479 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:31.372482 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:31.372499 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:31.374370 (+ 1871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:31.374828 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:31.374830 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:31.374832 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:31.374913 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:31.375165 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:31.375167 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:31.375169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:31.375369 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:31.376720 (+ 1351us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:31.376798 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 19:12:31.376916 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":117,"threads_started":1}
I20260620 19:12:31.377888 23288 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46103ef2a93e4a3e808e01f82d3273d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:12:31.380329 23364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e90aa5a6b5f14ea09f9f4ac55f55a7b5: Bootstrap starting.
I20260620 19:12:31.381646 23293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:31.370342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57520 (local address 127.9.155.125:42505)
0620 19:12:31.370445 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:31.370449 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:31.371910 (+ 1461us) server_negotiation.cc:407] Connection header received
0620 19:12:31.372367 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:31.372370 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:31.372417 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:31.372463 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:31.373997 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:31.374734 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:31.375846 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:31.376177 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:31.376511 (+ 334us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:31.378180 (+ 1669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:31.378184 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:31.378186 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:31.378213 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:31.379796 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:31.380331 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:31.380335 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:31.380336 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:31.380418 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:31.380719 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:31.380722 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:31.380724 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:31.380924 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:31.381034 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:31.381395 (+ 361us) server_negotiation.cc:299] Negotiation successful
0620 19:12:31.381509 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 19:12:31.382676 23380 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46103ef2a93e4a3e808e01f82d3273d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fcc0a38b4834220ad1fded011f2da88" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 45461 } } peers { permanent_uuid: "46103ef2a93e4a3e808e01f82d3273d6" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 42505 } } peers { permanent_uuid: "e90aa5a6b5f14ea09f9f4ac55f55a7b5" member_type: VOTER last_known_addr { host: "127.9.155.124" port: 45499 } } }
I20260620 19:12:31.382898 23380 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46103ef2a93e4a3e808e01f82d3273d6 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:12:31.383409 23364 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e90aa5a6b5f14ea09f9f4ac55f55a7b5: Neither blocks nor log segments found. Creating new log.
W20260620 19:12:31.383508 23166 tablet.cc:2369] T 00000000000000000000000000000000 P 4fcc0a38b4834220ad1fded011f2da88: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:12:31.380447 23145 master_runner.cc:186] Error getting master registration for 127.9.155.125:42505: OK, instance_id { permanent_uuid: "46103ef2a93e4a3e808e01f82d3273d6" instance_seqno: 1781982751090872 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 19:12:31.384385 23364 log.cc:824] T 00000000000000000000000000000000 P e90aa5a6b5f14ea09f9f4ac55f55a7b5: Log is configured to *not* fsync() on all Append() calls
I20260620 19:12:31.384989 23145 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 19:12:31.386627 23403 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 46103ef2a93e4a3e808e01f82d3273d6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:12:31.386947 23403 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:12:31.386945 23381 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fcc0a38b4834220ad1fded011f2da88 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fcc0a38b4834220ad1fded011f2da88" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 45461 } } peers { permanent_uuid: "46103ef2a93e4a3e808e01f82d3273d6" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 42505 } } peers { permanent_uuid: "e90aa5a6b5f14ea09f9f4ac55f55a7b5" member_type: VOTER last_known_addr { host: "127.9.155.124" port: 45499 } } }
I20260620 19:12:31.387157 23381 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fcc0a38b4834220ad1fded011f2da88 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:12:31.384653 23220 master_runner.cc:186] Error getting master registration for 127.9.155.124:45499: OK, instance_id { permanent_uuid: "e90aa5a6b5f14ea09f9f4ac55f55a7b5" instance_seqno: 1781982751317941 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 19:12:31.389603 23220 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 19:12:31.390076 23217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:31.370285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57794 (local address 127.9.155.126:45461)
0620 19:12:31.370395 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:31.370399 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:31.371571 (+ 1172us) server_negotiation.cc:407] Connection header received
0620 19:12:31.371777 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:31.371781 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:31.371838 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:31.371922 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:31.373394 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:31.374145 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:31.375404 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:31.375741 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:31.375824 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:31.385877 (+ 10053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:31.385882 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:31.385884 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:31.385910 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:31.387950 (+ 2040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:31.388481 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:31.388485 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:31.388487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:31.388570 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:31.388842 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:31.388846 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:31.388847 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:31.389097 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:31.389720 (+ 623us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:31.389807 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 19:12:31.389920 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 19:12:31.392176 23369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:31.371132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36606 (local address 127.9.155.124:45499)
0620 19:12:31.371253 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:31.371257 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:31.372173 (+ 916us) server_negotiation.cc:407] Connection header received
0620 19:12:31.372264 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:31.372267 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:31.372319 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:31.372398 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:31.377115 (+ 4717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:31.377616 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:31.386799 (+ 9183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:31.387122 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:31.387245 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:31.388585 (+ 1340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:31.388589 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:31.388592 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:31.388615 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:31.390155 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:31.390567 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:31.390570 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:31.390572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:31.390658 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:31.390933 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:31.390936 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:31.390938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:31.391139 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:31.391287 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:31.391908 (+ 621us) server_negotiation.cc:299] Negotiation successful
0620 19:12:31.392029 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 19:12:31.394203 23364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e90aa5a6b5f14ea09f9f4ac55f55a7b5: No bootstrap required, opened a new log
W20260620 19:12:31.399262 23392 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4fcc0a38b4834220ad1fded011f2da88: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:12:31.399358 23392 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:12:31.402107 23364 raft_consensus.cc:348] T 00000000000000000000000000000000 P e90aa5a6b5f14ea09f9f4ac55f55a7b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fcc0a38b4834220ad1fded011f2da88" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 45461 } } peers { permanent_uuid: "46103ef2a93e4a3e808e01f82d3273d6" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 42505 } } peers { permanent_uuid: "e90aa5a6b5f14ea09f9f4ac55f55a7b5" member_type: VOTER last_known_addr { host: "127.9.155.124" port: 45499 } }
I20260620 19:12:31.402420 23364 raft_consensus.cc:374] T 00000000000000000000000000000000 P e90aa5a6b5f14ea09f9f4ac55f55a7b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:12:31.402541 23364 raft_consensus.cc:729] T 00000000000000000000000000000000 P e90aa5a6b5f14ea09f9f4ac55f55a7b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e90aa5a6b5f14ea09f9f4ac55f55a7b5, State: Initialized, Role: FOLLOWER
I20260620 19:12:31.403081 23364 consensus_queue.cc:260] T 00000000000000000000000000000000 P e90aa5a6b5f14ea09f9f4ac55f55a7b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fcc0a38b4834220ad1fded011f2da88" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 45461 } } peers { permanent_uuid: "46103ef2a93e4a3e808e01f82d3273d6" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 42505 } } peers { permanent_uuid: "e90aa5a6b5f14ea09f9f4ac55f55a7b5" member_type: VOTER last_known_addr { host: "127.9.155.124" port: 45499 } }
I20260620 19:12:31.404647 23364 sys_catalog.cc:564] T 00000000000000000000000000000000 P e90aa5a6b5f14ea09f9f4ac55f55a7b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:12:31.405642 23407 sys_catalog.cc:455] T 00000000000000000000000000000000 P e90aa5a6b5f14ea09f9f4ac55f55a7b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fcc0a38b4834220ad1fded011f2da88" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 45461 } } peers { permanent_uuid: "46103ef2a93e4a3e808e01f82d3273d6" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 42505 } } peers { permanent_uuid: "e90aa5a6b5f14ea09f9f4ac55f55a7b5" member_type: VOTER last_known_addr { host: "127.9.155.124" port: 45499 } } }
I20260620 19:12:31.405809 23407 sys_catalog.cc:458] T 00000000000000000000000000000000 P e90aa5a6b5f14ea09f9f4ac55f55a7b5 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 19:12:31.411963 23418 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e90aa5a6b5f14ea09f9f4ac55f55a7b5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:12:31.412106 23418 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:12:31.413472 23294 master_runner.cc:263] No leader master found from master e90aa5a6b5f14ea09f9f4ac55f55a7b5
I20260620 19:12:31.413539 23294 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 19:12:31.420164 23381 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4fcc0a38b4834220ad1fded011f2da88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:12:31.420382 23381 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4fcc0a38b4834220ad1fded011f2da88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fcc0a38b4834220ad1fded011f2da88" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 45461 } } peers { permanent_uuid: "46103ef2a93e4a3e808e01f82d3273d6" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 42505 } } peers { permanent_uuid: "e90aa5a6b5f14ea09f9f4ac55f55a7b5" member_type: VOTER last_known_addr { host: "127.9.155.124" port: 45499 } }
I20260620 19:12:31.421373 23381 leader_election.cc:288] T 00000000000000000000000000000000 P 4fcc0a38b4834220ad1fded011f2da88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46103ef2a93e4a3e808e01f82d3273d6 (127.9.155.125:42505), e90aa5a6b5f14ea09f9f4ac55f55a7b5 (127.9.155.124:45499)
I20260620 19:12:31.422004 23338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4fcc0a38b4834220ad1fded011f2da88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e90aa5a6b5f14ea09f9f4ac55f55a7b5" is_pre_election: true
I20260620 19:12:31.422286 23338 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e90aa5a6b5f14ea09f9f4ac55f55a7b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fcc0a38b4834220ad1fded011f2da88 in term 0.
I20260620 19:12:31.422866 23162 leader_election.cc:302] T 00000000000000000000000000000000 P 4fcc0a38b4834220ad1fded011f2da88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fcc0a38b4834220ad1fded011f2da88, e90aa5a6b5f14ea09f9f4ac55f55a7b5; no voters:
I20260620 19:12:31.423539 23262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4fcc0a38b4834220ad1fded011f2da88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46103ef2a93e4a3e808e01f82d3273d6" is_pre_election: true
I20260620 19:12:31.423835 23262 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 46103ef2a93e4a3e808e01f82d3273d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fcc0a38b4834220ad1fded011f2da88 in term 0.
I20260620 19:12:31.424779 23381 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4fcc0a38b4834220ad1fded011f2da88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:12:31.424872 23381 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4fcc0a38b4834220ad1fded011f2da88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:12:31.424912 23381 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4fcc0a38b4834220ad1fded011f2da88 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:12:31.426192 23381 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4fcc0a38b4834220ad1fded011f2da88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fcc0a38b4834220ad1fded011f2da88" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 45461 } } peers { permanent_uuid: "46103ef2a93e4a3e808e01f82d3273d6" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 42505 } } peers { permanent_uuid: "e90aa5a6b5f14ea09f9f4ac55f55a7b5" member_type: VOTER last_known_addr { host: "127.9.155.124" port: 45499 } }
I20260620 19:12:31.426908 23262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4fcc0a38b4834220ad1fded011f2da88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46103ef2a93e4a3e808e01f82d3273d6"
I20260620 19:12:31.427028 23262 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46103ef2a93e4a3e808e01f82d3273d6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:12:31.429062 23262 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 46103ef2a93e4a3e808e01f82d3273d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fcc0a38b4834220ad1fded011f2da88 in term 1.
I20260620 19:12:31.429472 23162 leader_election.cc:302] T 00000000000000000000000000000000 P 4fcc0a38b4834220ad1fded011f2da88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46103ef2a93e4a3e808e01f82d3273d6, 4fcc0a38b4834220ad1fded011f2da88; no voters:
I20260620 19:12:31.429930 23381 leader_election.cc:288] T 00000000000000000000000000000000 P 4fcc0a38b4834220ad1fded011f2da88 [CANDIDATE]: Term 1 election: Requested vote from peers 46103ef2a93e4a3e808e01f82d3273d6 (127.9.155.125:42505), e90aa5a6b5f14ea09f9f4ac55f55a7b5 (127.9.155.124:45499)
I20260620 19:12:31.430397 23338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4fcc0a38b4834220ad1fded011f2da88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e90aa5a6b5f14ea09f9f4ac55f55a7b5"
I20260620 19:12:31.430529 23338 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e90aa5a6b5f14ea09f9f4ac55f55a7b5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:12:31.431883 23338 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e90aa5a6b5f14ea09f9f4ac55f55a7b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fcc0a38b4834220ad1fded011f2da88 in term 1.
I20260620 19:12:31.434998 23381 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4fcc0a38b4834220ad1fded011f2da88 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:12:31.435151 23381 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4fcc0a38b4834220ad1fded011f2da88 [term 1 LEADER]: Becoming Leader. State: Replica: 4fcc0a38b4834220ad1fded011f2da88, State: Running, Role: LEADER
I20260620 19:12:31.435650 23381 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4fcc0a38b4834220ad1fded011f2da88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fcc0a38b4834220ad1fded011f2da88" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 45461 } } peers { permanent_uuid: "46103ef2a93e4a3e808e01f82d3273d6" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 42505 } } peers { permanent_uuid: "e90aa5a6b5f14ea09f9f4ac55f55a7b5" member_type: VOTER last_known_addr { host: "127.9.155.124" port: 45499 } }
I20260620 19:12:31.437604 23381 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fcc0a38b4834220ad1fded011f2da88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4fcc0a38b4834220ad1fded011f2da88. Latest consensus state: current_term: 1 leader_uuid: "4fcc0a38b4834220ad1fded011f2da88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fcc0a38b4834220ad1fded011f2da88" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 45461 } } peers { permanent_uuid: "46103ef2a93e4a3e808e01f82d3273d6" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 42505 } } peers { permanent_uuid: "e90aa5a6b5f14ea09f9f4ac55f55a7b5" member_type: VOTER last_known_addr { host: "127.9.155.124" port: 45499 } } }
I20260620 19:12:31.437790 23381 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fcc0a38b4834220ad1fded011f2da88 [sys.catalog]: This master's current role is: LEADER
I20260620 19:12:31.438517 23422 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:12:31.439569 23422 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:12:31.449097 23338 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e90aa5a6b5f14ea09f9f4ac55f55a7b5 [term 1 FOLLOWER]: Refusing update from remote peer 4fcc0a38b4834220ad1fded011f2da88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:12:31.449867 23420 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4fcc0a38b4834220ad1fded011f2da88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e90aa5a6b5f14ea09f9f4ac55f55a7b5" member_type: VOTER last_known_addr { host: "127.9.155.124" port: 45499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:12:31.455003 23407 sys_catalog.cc:455] T 00000000000000000000000000000000 P e90aa5a6b5f14ea09f9f4ac55f55a7b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4fcc0a38b4834220ad1fded011f2da88. Latest consensus state: current_term: 1 leader_uuid: "4fcc0a38b4834220ad1fded011f2da88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fcc0a38b4834220ad1fded011f2da88" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 45461 } } peers { permanent_uuid: "46103ef2a93e4a3e808e01f82d3273d6" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 42505 } } peers { permanent_uuid: "e90aa5a6b5f14ea09f9f4ac55f55a7b5" member_type: VOTER last_known_addr { host: "127.9.155.124" port: 45499 } } }
I20260620 19:12:31.455129 23407 sys_catalog.cc:458] T 00000000000000000000000000000000 P e90aa5a6b5f14ea09f9f4ac55f55a7b5 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:12:31.458388 23262 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 46103ef2a93e4a3e808e01f82d3273d6 [term 1 FOLLOWER]: Refusing update from remote peer 4fcc0a38b4834220ad1fded011f2da88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:12:31.458979 23420 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4fcc0a38b4834220ad1fded011f2da88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46103ef2a93e4a3e808e01f82d3273d6" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 42505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:12:31.462965 23380 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46103ef2a93e4a3e808e01f82d3273d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4fcc0a38b4834220ad1fded011f2da88. Latest consensus state: current_term: 1 leader_uuid: "4fcc0a38b4834220ad1fded011f2da88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fcc0a38b4834220ad1fded011f2da88" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 45461 } } peers { permanent_uuid: "46103ef2a93e4a3e808e01f82d3273d6" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 42505 } } peers { permanent_uuid: "e90aa5a6b5f14ea09f9f4ac55f55a7b5" member_type: VOTER last_known_addr { host: "127.9.155.124" port: 45499 } } }
I20260620 19:12:31.463084 23380 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46103ef2a93e4a3e808e01f82d3273d6 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:12:31.467521 23420 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fcc0a38b4834220ad1fded011f2da88 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4fcc0a38b4834220ad1fded011f2da88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fcc0a38b4834220ad1fded011f2da88" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 45461 } } peers { permanent_uuid: "46103ef2a93e4a3e808e01f82d3273d6" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 42505 } } peers { permanent_uuid: "e90aa5a6b5f14ea09f9f4ac55f55a7b5" member_type: VOTER last_known_addr { host: "127.9.155.124" port: 45499 } } }
I20260620 19:12:31.467664 23420 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fcc0a38b4834220ad1fded011f2da88 [sys.catalog]: This master's current role is: LEADER
I20260620 19:12:31.471321 23380 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46103ef2a93e4a3e808e01f82d3273d6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4fcc0a38b4834220ad1fded011f2da88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fcc0a38b4834220ad1fded011f2da88" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 45461 } } peers { permanent_uuid: "46103ef2a93e4a3e808e01f82d3273d6" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 42505 } } peers { permanent_uuid: "e90aa5a6b5f14ea09f9f4ac55f55a7b5" member_type: VOTER last_known_addr { host: "127.9.155.124" port: 45499 } } }
I20260620 19:12:31.471433 23380 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46103ef2a93e4a3e808e01f82d3273d6 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:12:31.472086 23423 mvcc.cc:205] Tried to move back new op lower bound from 7299001349928054784 to 7299001349884137472. Current Snapshot: MvccSnapshot[applied={T|T < 7299001349928054784}]
I20260620 19:12:31.473642 23426 mvcc.cc:205] Tried to move back new op lower bound from 7299001349928054784 to 7299001349884137472. Current Snapshot: MvccSnapshot[applied={T|T < 7299001349928054784}]
I20260620 19:12:31.474859 23381 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fcc0a38b4834220ad1fded011f2da88 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4fcc0a38b4834220ad1fded011f2da88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fcc0a38b4834220ad1fded011f2da88" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 45461 } } peers { permanent_uuid: "46103ef2a93e4a3e808e01f82d3273d6" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 42505 } } peers { permanent_uuid: "e90aa5a6b5f14ea09f9f4ac55f55a7b5" member_type: VOTER last_known_addr { host: "127.9.155.124" port: 45499 } } }
I20260620 19:12:31.474992 23381 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fcc0a38b4834220ad1fded011f2da88 [sys.catalog]: This master's current role is: LEADER
I20260620 19:12:31.476290 23407 sys_catalog.cc:455] T 00000000000000000000000000000000 P e90aa5a6b5f14ea09f9f4ac55f55a7b5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4fcc0a38b4834220ad1fded011f2da88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fcc0a38b4834220ad1fded011f2da88" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 45461 } } peers { permanent_uuid: "46103ef2a93e4a3e808e01f82d3273d6" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 42505 } } peers { permanent_uuid: "e90aa5a6b5f14ea09f9f4ac55f55a7b5" member_type: VOTER last_known_addr { host: "127.9.155.124" port: 45499 } } }
I20260620 19:12:31.476408 23407 sys_catalog.cc:458] T 00000000000000000000000000000000 P e90aa5a6b5f14ea09f9f4ac55f55a7b5 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:12:31.479148 23424 mvcc.cc:205] Tried to move back new op lower bound from 7299001349928054784 to 7299001349884137472. Current Snapshot: MvccSnapshot[applied={T|T < 7299001349928054784}]
I20260620 19:12:31.482648 23422 catalog_manager.cc:1347] Generated new cluster ID: 9579c6e6ad2c41428e838a73571a2b1d
I20260620 19:12:31.482721 23422 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:12:31.552742 23422 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:12:31.554355 23422 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:12:31.576164 23422 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4fcc0a38b4834220ad1fded011f2da88: Generated new TSK 0
I20260620 19:12:31.577303 23422 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:12:32.388345 23403 catalog_manager.cc:1259] Loaded cluster ID: 9579c6e6ad2c41428e838a73571a2b1d
I20260620 19:12:32.388432 23403 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 46103ef2a93e4a3e808e01f82d3273d6: loading cluster ID for follower catalog manager: success
I20260620 19:12:32.390130 23403 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 46103ef2a93e4a3e808e01f82d3273d6: acquiring CA information for follower catalog manager: success
I20260620 19:12:32.391345 23403 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 46103ef2a93e4a3e808e01f82d3273d6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 19:12:32.413395 23418 catalog_manager.cc:1259] Loaded cluster ID: 9579c6e6ad2c41428e838a73571a2b1d
I20260620 19:12:32.413483 23418 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e90aa5a6b5f14ea09f9f4ac55f55a7b5: loading cluster ID for follower catalog manager: success
I20260620 19:12:32.415186 23418 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e90aa5a6b5f14ea09f9f4ac55f55a7b5: acquiring CA information for follower catalog manager: success
I20260620 19:12:32.416420 23418 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e90aa5a6b5f14ea09f9f4ac55f55a7b5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 19:12:32.639386 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 23145
I20260620 19:12:32.644963 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 23220
I20260620 19:12:32.649798 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 23294
2026-06-20T19:12:32Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3035 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982595231230-9837-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:12:32 dist-test-slave-n3fx krb5kdc[23436](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:12:32 dist-test-slave-n3fx krb5kdc[23436](info): set up 2 sockets
Jun 20 19:12:32 dist-test-slave-n3fx krb5kdc[23436](info): commencing operation
krb5kdc: starting...
W20260620 19:12:33.604300 9837 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982595231230-9837-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982595231230-9837-0/krb5kdc/test-user.keytab.
Jun 20 19:12:33 dist-test-slave-n3fx krb5kdc[23436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:12:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:12:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.155.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.155.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.155.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab.
Entry for principal kudu/127.9.155.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab.
I20260620 19:12:33.723066 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982595231230-9837-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982595231230-9837-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982595231230-9837-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:43575
--webserver_interface=127.9.155.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab
--principal=kudu/127.9.155.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:41229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.155.126:43575
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:12:33.829944 23449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:33.830212 23449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:33.830252 23449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:33.834265 23449 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:12:33.834333 23449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:33.834352 23449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:33.834368 23449 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:12:33.834383 23449 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:12:33.839725 23449 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:41229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982595231230-9837-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982595231230-9837-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.155.126:43575
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab
--principal=kudu/127.9.155.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:43575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.155.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23449
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982595231230-9837-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:12:33.840794 23449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:33.841769 23449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:12:33.850097 23455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:33.850108 23454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:33.850142 23457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:33.850533 23449 server_base.cc:1034] running on GCE node
I20260620 19:12:33.851274 23449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:33.852577 23449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:33.853788 23449 hybrid_clock.cc:648] HybridClock initialized: now 1781982753853774 us; error 35 us; skew 500 ppm
Jun 20 19:12:33 dist-test-slave-n3fx krb5kdc[23436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982753, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:33.856572 23449 init.cc:377] Logged in from keytab as kudu/127.9.155.126@KRBTEST.COM (short username kudu)
I20260620 19:12:33.857455 23449 webserver.cc:460] Webserver started at http://127.9.155.126:34777/ using document root <none> and password file <none>
I20260620 19:12:33.857990 23449 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:33.858055 23449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:12:33.858253 23449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:12:33.860337 23449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982595231230-9837-0/minicluster-data/master-0/data/instance:
uuid: "1a7b7c8385744c64aa769ebc0a958ff5"
format_stamp: "Formatted at 2026-06-20 19:12:33 on dist-test-slave-n3fx"
server_key: "f226bb2f897a92f8baff2ab3393a6831"
server_key_iv: "98ba2fd27123b418c863d732c06f75bb"
server_key_version: "encryptionkey@0"
I20260620 19:12:33.860872 23449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982595231230-9837-0/minicluster-data/master-0/wal/instance:
uuid: "1a7b7c8385744c64aa769ebc0a958ff5"
format_stamp: "Formatted at 2026-06-20 19:12:33 on dist-test-slave-n3fx"
server_key: "f226bb2f897a92f8baff2ab3393a6831"
server_key_iv: "98ba2fd27123b418c863d732c06f75bb"
server_key_version: "encryptionkey@0"
I20260620 19:12:33.864451 23449 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:12:33.866761 23464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:33.867950 23449 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 19:12:33.868060 23449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982595231230-9837-0/minicluster-data/master-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982595231230-9837-0/minicluster-data/master-0/wal
uuid: "1a7b7c8385744c64aa769ebc0a958ff5"
format_stamp: "Formatted at 2026-06-20 19:12:33 on dist-test-slave-n3fx"
server_key: "f226bb2f897a92f8baff2ab3393a6831"
server_key_iv: "98ba2fd27123b418c863d732c06f75bb"
server_key_version: "encryptionkey@0"
I20260620 19:12:33.868155 23449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982595231230-9837-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982595231230-9837-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982595231230-9837-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:33.883422 23449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:33.886358 23449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:33.886529 23449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:33.893307 23449 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.126:43575
I20260620 19:12:33.893328 23516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.126:43575 every 8 connection(s)
I20260620 19:12:33.894187 23449 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
I20260620 19:12:33.896852 23517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:12:33.898906 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 23449
I20260620 19:12:33.898998 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982595231230-9837-0/minicluster-data/master-0/wal/instance
I20260620 19:12:33.899240 9837 external_mini_cluster.cc:1428] Setting key d80c9105a350b8d290d500991310421b
I20260620 19:12:33.902848 23517 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a7b7c8385744c64aa769ebc0a958ff5: Bootstrap starting.
I20260620 19:12:33.905380 23517 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a7b7c8385744c64aa769ebc0a958ff5: Neither blocks nor log segments found. Creating new log.
I20260620 19:12:33.906206 23517 log.cc:824] T 00000000000000000000000000000000 P 1a7b7c8385744c64aa769ebc0a958ff5: Log is configured to *not* fsync() on all Append() calls
Jun 20 19:12:33 dist-test-slave-n3fx krb5kdc[23436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:12:33.908043 23517 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a7b7c8385744c64aa769ebc0a958ff5: No bootstrap required, opened a new log
I20260620 19:12:33.911489 23517 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a7b7c8385744c64aa769ebc0a958ff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a7b7c8385744c64aa769ebc0a958ff5" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 43575 } }
I20260620 19:12:33.911701 23517 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a7b7c8385744c64aa769ebc0a958ff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:12:33.911772 23517 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a7b7c8385744c64aa769ebc0a958ff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a7b7c8385744c64aa769ebc0a958ff5, State: Initialized, Role: FOLLOWER
I20260620 19:12:33.912237 23517 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a7b7c8385744c64aa769ebc0a958ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a7b7c8385744c64aa769ebc0a958ff5" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 43575 } }
I20260620 19:12:33.912382 23517 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a7b7c8385744c64aa769ebc0a958ff5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:12:33.912456 23517 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a7b7c8385744c64aa769ebc0a958ff5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:12:33.912547 23517 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a7b7c8385744c64aa769ebc0a958ff5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:12:33.912936 23520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:33.900525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40852 (local address 127.9.155.126:43575)
0620 19:12:33.900921 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:33.900939 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:33.900981 (+ 42us) server_negotiation.cc:407] Connection header received
0620 19:12:33.901778 (+ 797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:33.901801 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:33.902203 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:33.902552 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:33.903163 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:33.904649 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:33.905449 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:33.906121 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:33.906266 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:33.908567 (+ 2301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:33.908583 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:33.908596 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:33.908621 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:33.910181 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:33.910584 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:33.910589 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:33.910593 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:33.910705 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:33.910952 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:33.910956 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:33.910957 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:33.911327 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:33.911509 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:33.911810 (+ 301us) server_negotiation.cc:299] Negotiation successful
0620 19:12:33.912094 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":98,"threads_started":1}
I20260620 19:12:33.913448 23517 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a7b7c8385744c64aa769ebc0a958ff5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a7b7c8385744c64aa769ebc0a958ff5" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 43575 } }
I20260620 19:12:33.913761 23517 leader_election.cc:302] T 00000000000000000000000000000000 P 1a7b7c8385744c64aa769ebc0a958ff5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a7b7c8385744c64aa769ebc0a958ff5; no voters:
I20260620 19:12:33.914027 23517 leader_election.cc:288] T 00000000000000000000000000000000 P 1a7b7c8385744c64aa769ebc0a958ff5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:12:33.914141 23522 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a7b7c8385744c64aa769ebc0a958ff5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:12:33.914328 23522 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a7b7c8385744c64aa769ebc0a958ff5 [term 1 LEADER]: Becoming Leader. State: Replica: 1a7b7c8385744c64aa769ebc0a958ff5, State: Running, Role: LEADER
I20260620 19:12:33.914688 23522 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a7b7c8385744c64aa769ebc0a958ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a7b7c8385744c64aa769ebc0a958ff5" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 43575 } }
I20260620 19:12:33.915227 23517 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a7b7c8385744c64aa769ebc0a958ff5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:12:33.916000 9837 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:12:33.916247 23524 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a7b7c8385744c64aa769ebc0a958ff5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a7b7c8385744c64aa769ebc0a958ff5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a7b7c8385744c64aa769ebc0a958ff5" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 43575 } } }
I20260620 19:12:33.916354 23524 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a7b7c8385744c64aa769ebc0a958ff5 [sys.catalog]: This master's current role is: LEADER
I20260620 19:12:33.916755 23531 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:12:33.917630 23523 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a7b7c8385744c64aa769ebc0a958ff5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a7b7c8385744c64aa769ebc0a958ff5. Latest consensus state: current_term: 1 leader_uuid: "1a7b7c8385744c64aa769ebc0a958ff5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a7b7c8385744c64aa769ebc0a958ff5" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 43575 } } }
I20260620 19:12:33.917737 23523 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a7b7c8385744c64aa769ebc0a958ff5 [sys.catalog]: This master's current role is: LEADER
I20260620 19:12:33.919837 23531 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:12:33.925624 23531 catalog_manager.cc:1347] Generated new cluster ID: 60e3a50535bd4a26952b999a3d2d9252
I20260620 19:12:33.925704 23531 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:12:33.944707 23531 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:12:33.946098 23531 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:12:33.952327 23531 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a7b7c8385744c64aa769ebc0a958ff5: Generated new TSK 0
I20260620 19:12:33.953030 23531 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:12:33.972090 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 23449
2026-06-20T19:12:33Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1321 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:12:34 dist-test-slave-n3fx krb5kdc[23544](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:12:34 dist-test-slave-n3fx krb5kdc[23544](info): set up 2 sockets
Jun 20 19:12:34 dist-test-slave-n3fx krb5kdc[23544](info): commencing operation
krb5kdc: starting...
W20260620 19:12:34.936149 9837 mini_kdc.cc:121] Time spent starting KDC: real 0.945s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/krb5kdc/test-user.keytab.
Jun 20 19:12:35 dist-test-slave-n3fx krb5kdc[23544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982755, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:12:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:12:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.155.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.155.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.155.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab.
Entry for principal kudu/127.9.155.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab.
I20260620 19:12:35.059340 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:33551
--webserver_interface=127.9.155.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab
--principal=kudu/127.9.155.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:36051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.155.126:33551
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:12:35.169057 23557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:35.169376 23557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:35.169438 23557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:35.173713 23557 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:12:35.173799 23557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:35.173826 23557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:35.173849 23557 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:12:35.173874 23557 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:12:35.179502 23557 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:36051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.155.126:33551
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab
--principal=kudu/127.9.155.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:33551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.155.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23557
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:12:35.180649 23557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:35.181685 23557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:12:35.190671 23565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:35.190730 23562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:35.190675 23563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:35.191056 23557 server_base.cc:1034] running on GCE node
I20260620 19:12:35.191655 23557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:35.192878 23557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:35.194048 23557 hybrid_clock.cc:648] HybridClock initialized: now 1781982755194036 us; error 30 us; skew 500 ppm
Jun 20 19:12:35 dist-test-slave-n3fx krb5kdc[23544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982755, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:35.201519 23557 init.cc:377] Logged in from keytab as kudu/127.9.155.126@KRBTEST.COM (short username kudu)
I20260620 19:12:35.202464 23557 webserver.cc:460] Webserver started at http://127.9.155.126:40697/ using document root <none> and password file <none>
I20260620 19:12:35.203009 23557 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:35.203070 23557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:12:35.203243 23557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:12:35.205406 23557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/master-0/data/instance:
uuid: "b34ec788c4684b4bb78bf6fed9657a8c"
format_stamp: "Formatted at 2026-06-20 19:12:35 on dist-test-slave-n3fx"
server_key: "4e368b3c54979a8b4297115bc51f9dcd"
server_key_iv: "cbad7a6c1833629c473316f986ab7d22"
server_key_version: "encryptionkey@0"
I20260620 19:12:35.205945 23557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/master-0/wal/instance:
uuid: "b34ec788c4684b4bb78bf6fed9657a8c"
format_stamp: "Formatted at 2026-06-20 19:12:35 on dist-test-slave-n3fx"
server_key: "4e368b3c54979a8b4297115bc51f9dcd"
server_key_iv: "cbad7a6c1833629c473316f986ab7d22"
server_key_version: "encryptionkey@0"
I20260620 19:12:35.209770 23557 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 19:12:35.212502 23572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:35.213860 23557 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:12:35.213982 23557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/master-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/master-0/wal
uuid: "b34ec788c4684b4bb78bf6fed9657a8c"
format_stamp: "Formatted at 2026-06-20 19:12:35 on dist-test-slave-n3fx"
server_key: "4e368b3c54979a8b4297115bc51f9dcd"
server_key_iv: "cbad7a6c1833629c473316f986ab7d22"
server_key_version: "encryptionkey@0"
I20260620 19:12:35.214107 23557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:35.223523 23557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:35.226768 23557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:35.226948 23557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:35.233632 23557 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.126:33551
I20260620 19:12:35.233930 23624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.126:33551 every 8 connection(s)
I20260620 19:12:35.234592 23557 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
I20260620 19:12:35.235540 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 23557
I20260620 19:12:35.235641 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/master-0/wal/instance
I20260620 19:12:35.235898 9837 external_mini_cluster.cc:1428] Setting key 641ca1167ebdb0a168bd3b71ef35b7e7
I20260620 19:12:35.237743 23625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:12:35.244768 23625 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b34ec788c4684b4bb78bf6fed9657a8c: Bootstrap starting.
Jun 20 19:12:35 dist-test-slave-n3fx krb5kdc[23544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982755, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:12:35.247550 23625 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b34ec788c4684b4bb78bf6fed9657a8c: Neither blocks nor log segments found. Creating new log.
I20260620 19:12:35.248466 23625 log.cc:824] T 00000000000000000000000000000000 P b34ec788c4684b4bb78bf6fed9657a8c: Log is configured to *not* fsync() on all Append() calls
I20260620 19:12:35.250842 23625 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b34ec788c4684b4bb78bf6fed9657a8c: No bootstrap required, opened a new log
I20260620 19:12:35.251225 23628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:35.237286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51826 (local address 127.9.155.126:33551)
0620 19:12:35.237703 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:35.237721 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:35.237764 (+ 43us) server_negotiation.cc:407] Connection header received
0620 19:12:35.238587 (+ 823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:35.238622 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:35.239043 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:35.239446 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:35.240125 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:35.241691 (+ 1566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:35.242634 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:35.243316 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:35.243484 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:35.246105 (+ 2621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:35.246112 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:35.246132 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:35.246167 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:35.248063 (+ 1896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:35.248558 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:35.248564 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:35.248570 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:35.248717 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:35.249005 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:35.249008 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:35.249009 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:35.249483 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:35.249686 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:35.249927 (+ 241us) server_negotiation.cc:299] Negotiation successful
0620 19:12:35.250254 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":109,"threads_started":1}
I20260620 19:12:35.255072 23625 raft_consensus.cc:348] T 00000000000000000000000000000000 P b34ec788c4684b4bb78bf6fed9657a8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b34ec788c4684b4bb78bf6fed9657a8c" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 33551 } }
I20260620 19:12:35.255324 23625 raft_consensus.cc:374] T 00000000000000000000000000000000 P b34ec788c4684b4bb78bf6fed9657a8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:12:35.255360 23625 raft_consensus.cc:729] T 00000000000000000000000000000000 P b34ec788c4684b4bb78bf6fed9657a8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b34ec788c4684b4bb78bf6fed9657a8c, State: Initialized, Role: FOLLOWER
I20260620 19:12:35.255729 23625 consensus_queue.cc:260] T 00000000000000000000000000000000 P b34ec788c4684b4bb78bf6fed9657a8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b34ec788c4684b4bb78bf6fed9657a8c" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 33551 } }
I20260620 19:12:35.255841 23625 raft_consensus.cc:388] T 00000000000000000000000000000000 P b34ec788c4684b4bb78bf6fed9657a8c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:12:35.255878 23625 raft_consensus.cc:482] T 00000000000000000000000000000000 P b34ec788c4684b4bb78bf6fed9657a8c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:12:35.255946 23625 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b34ec788c4684b4bb78bf6fed9657a8c [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:12:35.257000 23625 raft_consensus.cc:504] T 00000000000000000000000000000000 P b34ec788c4684b4bb78bf6fed9657a8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b34ec788c4684b4bb78bf6fed9657a8c" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 33551 } }
I20260620 19:12:35.257325 23625 leader_election.cc:302] T 00000000000000000000000000000000 P b34ec788c4684b4bb78bf6fed9657a8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b34ec788c4684b4bb78bf6fed9657a8c; no voters:
I20260620 19:12:35.257593 23625 leader_election.cc:288] T 00000000000000000000000000000000 P b34ec788c4684b4bb78bf6fed9657a8c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:12:35.257766 23630 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b34ec788c4684b4bb78bf6fed9657a8c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:12:35.257969 23630 raft_consensus.cc:686] T 00000000000000000000000000000000 P b34ec788c4684b4bb78bf6fed9657a8c [term 1 LEADER]: Becoming Leader. State: Replica: b34ec788c4684b4bb78bf6fed9657a8c, State: Running, Role: LEADER
I20260620 19:12:35.258265 23630 consensus_queue.cc:237] T 00000000000000000000000000000000 P b34ec788c4684b4bb78bf6fed9657a8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b34ec788c4684b4bb78bf6fed9657a8c" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 33551 } }
I20260620 19:12:35.258769 23625 sys_catalog.cc:564] T 00000000000000000000000000000000 P b34ec788c4684b4bb78bf6fed9657a8c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:12:35.260039 23632 sys_catalog.cc:455] T 00000000000000000000000000000000 P b34ec788c4684b4bb78bf6fed9657a8c [sys.catalog]: SysCatalogTable state changed. Reason: New leader b34ec788c4684b4bb78bf6fed9657a8c. Latest consensus state: current_term: 1 leader_uuid: "b34ec788c4684b4bb78bf6fed9657a8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b34ec788c4684b4bb78bf6fed9657a8c" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 33551 } } }
I20260620 19:12:35.260071 23631 sys_catalog.cc:455] T 00000000000000000000000000000000 P b34ec788c4684b4bb78bf6fed9657a8c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b34ec788c4684b4bb78bf6fed9657a8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b34ec788c4684b4bb78bf6fed9657a8c" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 33551 } } }
I20260620 19:12:35.260192 23632 sys_catalog.cc:458] T 00000000000000000000000000000000 P b34ec788c4684b4bb78bf6fed9657a8c [sys.catalog]: This master's current role is: LEADER
I20260620 19:12:35.260192 23631 sys_catalog.cc:458] T 00000000000000000000000000000000 P b34ec788c4684b4bb78bf6fed9657a8c [sys.catalog]: This master's current role is: LEADER
I20260620 19:12:35.260668 23639 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:12:35.263827 23639 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:12:35.269691 23639 catalog_manager.cc:1347] Generated new cluster ID: d9a35173f7a344e9b05224cb361a8f31
I20260620 19:12:35.269866 23639 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:12:35.284302 23639 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:12:35.285372 23639 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:12:35.292625 23639 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b34ec788c4684b4bb78bf6fed9657a8c: Generated new TSK 0
I20260620 19:12:35.293398 23639 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.155.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.155.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.155.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.65.keytab.
Entry for principal kudu/127.9.155.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.65.keytab.
I20260620 19:12:35.324968 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.65:0
--local_ip_for_outbound_sockets=127.9.155.65
--webserver_interface=127.9.155.65
--webserver_port=0
--tserver_master_addrs=127.9.155.126:33551
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.65.keytab
--principal=kudu/127.9.155.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:36051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:12:35.445741 23651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:35.446050 23651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:35.446096 23651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:35.450475 23651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:35.450572 23651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:35.450635 23651 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.65
I20260620 19:12:35.456465 23651 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:36051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.65.keytab
--principal=kudu/127.9.155.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.155.65
--webserver_port=0
--tserver_master_addrs=127.9.155.126:33551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23651
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.65
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:12:35.457737 23651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:35.458822 23651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:12:35.469163 23659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:35.469170 23656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:35.469296 23651 server_base.cc:1034] running on GCE node
W20260620 19:12:35.469501 23657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:35.469947 23651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:35.470650 23651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:35.471810 23651 hybrid_clock.cc:648] HybridClock initialized: now 1781982755471778 us; error 50 us; skew 500 ppm
Jun 20 19:12:35 dist-test-slave-n3fx krb5kdc[23544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982755, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:35.474771 23651 init.cc:377] Logged in from keytab as kudu/127.9.155.65@KRBTEST.COM (short username kudu)
I20260620 19:12:35.475782 23651 webserver.cc:460] Webserver started at http://127.9.155.65:40617/ using document root <none> and password file <none>
I20260620 19:12:35.476382 23651 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:35.476449 23651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:12:35.476670 23651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:12:35.478755 23651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-0/data/instance:
uuid: "2b13fa8f898b4c99b52c87c89708aa37"
format_stamp: "Formatted at 2026-06-20 19:12:35 on dist-test-slave-n3fx"
server_key: "6a53c726ff4021dba8d7ac3f9460e48a"
server_key_iv: "ab9796f5b048326491fe0833c8761fd0"
server_key_version: "encryptionkey@0"
I20260620 19:12:35.479290 23651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-0/wal/instance:
uuid: "2b13fa8f898b4c99b52c87c89708aa37"
format_stamp: "Formatted at 2026-06-20 19:12:35 on dist-test-slave-n3fx"
server_key: "6a53c726ff4021dba8d7ac3f9460e48a"
server_key_iv: "ab9796f5b048326491fe0833c8761fd0"
server_key_version: "encryptionkey@0"
I20260620 19:12:35.483388 23651 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260620 19:12:35.486207 23666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:35.487632 23651 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 19:12:35.487758 23651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-0/wal
uuid: "2b13fa8f898b4c99b52c87c89708aa37"
format_stamp: "Formatted at 2026-06-20 19:12:35 on dist-test-slave-n3fx"
server_key: "6a53c726ff4021dba8d7ac3f9460e48a"
server_key_iv: "ab9796f5b048326491fe0833c8761fd0"
server_key_version: "encryptionkey@0"
I20260620 19:12:35.487896 23651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:35.503830 23651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:35.507225 23651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:35.507411 23651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:35.508049 23651 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:12:35.508963 23651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:12:35.509013 23651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:35.509056 23651 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:12:35.509084 23651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:35.519192 23651 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.65:46309
I20260620 19:12:35.519330 23779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.65:46309 every 8 connection(s)
I20260620 19:12:35.520207 23651 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
I20260620 19:12:35.522632 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 23651
I20260620 19:12:35.522768 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-0/wal/instance
I20260620 19:12:35.523097 9837 external_mini_cluster.cc:1428] Setting key 4079ed0cd56a0bf182fd8615be4acea0
Jun 20 19:12:35 dist-test-slave-n3fx krb5kdc[23544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982755, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.65@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.155.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.155.66@KRBTEST.COM" created.
I20260620 19:12:35.535282 23628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:35.522204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.65:45867 (local address 127.9.155.126:33551)
0620 19:12:35.522361 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:35.522365 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:35.523574 (+ 1209us) server_negotiation.cc:407] Connection header received
0620 19:12:35.525258 (+ 1684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:35.525261 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:35.525316 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:35.525391 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:35.526637 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:35.527178 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:35.528391 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:35.528772 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:35.528897 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:35.531605 (+ 2708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:35.531609 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:35.531612 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:35.531658 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:35.533111 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:35.533634 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:35.533638 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:35.533640 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:35.533722 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:35.534086 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:35.534090 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:35.534092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:35.534304 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:35.534391 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:35.534968 (+ 577us) server_negotiation.cc:299] Negotiation successful
0620 19:12:35.535108 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 19:12:35.536170 23782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:35.522714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:33551 (local address 127.9.155.65:45867)
0620 19:12:35.523328 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:35.523378 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:35.525029 (+ 1651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:35.525603 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:35.525611 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:35.525984 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:35.526477 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:35.526490 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:35.527299 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:35.527302 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:35.528245 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:35.528258 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:35.528949 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:35.528952 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:35.529115 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:35.529764 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:12:35.529783 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:35.531405 (+ 1622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:35.533232 (+ 1827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:35.533238 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:35.533257 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:35.533526 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:35.533810 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:35.533813 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:35.533815 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:35.533966 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:35.534404 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:35.534412 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:35.534714 (+ 302us) client_negotiation.cc:769] Sending connection context
0620 19:12:35.534915 (+ 201us) client_negotiation.cc:240] Negotiation successful
0620 19:12:35.535299 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":153,"threads_started":1}
I20260620 19:12:35.537598 23780 heartbeater.cc:347] Connected to a master server at 127.9.155.126:33551
I20260620 19:12:35.537843 23780 heartbeater.cc:464] Registering TS with master...
I20260620 19:12:35.538378 23780 heartbeater.cc:511] Master 127.9.155.126:33551 requested a full tablet report, sending...
I20260620 19:12:35.540122 23588 ts_manager.cc:194] Registered new tserver with Master: 2b13fa8f898b4c99b52c87c89708aa37 (127.9.155.65:46309)
I20260620 19:12:35.541747 23588 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.155.65@KRBTEST.COM'} at 127.9.155.65:45867
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.155.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.66.keytab.
Entry for principal kudu/127.9.155.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.66.keytab.
I20260620 19:12:35.552016 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.66:0
--local_ip_for_outbound_sockets=127.9.155.66
--webserver_interface=127.9.155.66
--webserver_port=0
--tserver_master_addrs=127.9.155.126:33551
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.66.keytab
--principal=kudu/127.9.155.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:36051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:12:35.662807 23785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:35.663086 23785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:35.663151 23785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:35.667253 23785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:35.667318 23785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:35.667403 23785 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.66
I20260620 19:12:35.672924 23785 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:36051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.66.keytab
--principal=kudu/127.9.155.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.155.66
--webserver_port=0
--tserver_master_addrs=127.9.155.126:33551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23785
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.66
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:12:35.674000 23785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:35.675027 23785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:12:35.684418 23790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:35.684423 23791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:35.684669 23785 server_base.cc:1034] running on GCE node
W20260620 19:12:35.684813 23793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:35.685206 23785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:35.685863 23785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:35.687026 23785 hybrid_clock.cc:648] HybridClock initialized: now 1781982755687005 us; error 33 us; skew 500 ppm
Jun 20 19:12:35 dist-test-slave-n3fx krb5kdc[23544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982755, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:35.689762 23785 init.cc:377] Logged in from keytab as kudu/127.9.155.66@KRBTEST.COM (short username kudu)
I20260620 19:12:35.690659 23785 webserver.cc:460] Webserver started at http://127.9.155.66:33871/ using document root <none> and password file <none>
I20260620 19:12:35.691223 23785 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:35.691286 23785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:12:35.691483 23785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:12:35.693609 23785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-1/data/instance:
uuid: "f70693a176b0410cac470f692003061a"
format_stamp: "Formatted at 2026-06-20 19:12:35 on dist-test-slave-n3fx"
server_key: "5f518b7d8c2a8e6aae061b4da68b21af"
server_key_iv: "ef32bf5b29d8a758a29972e0a86100dc"
server_key_version: "encryptionkey@0"
I20260620 19:12:35.694147 23785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-1/wal/instance:
uuid: "f70693a176b0410cac470f692003061a"
format_stamp: "Formatted at 2026-06-20 19:12:35 on dist-test-slave-n3fx"
server_key: "5f518b7d8c2a8e6aae061b4da68b21af"
server_key_iv: "ef32bf5b29d8a758a29972e0a86100dc"
server_key_version: "encryptionkey@0"
I20260620 19:12:35.698069 23785 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 19:12:35.700590 23800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:35.701956 23785 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:12:35.702085 23785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-1/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-1/wal
uuid: "f70693a176b0410cac470f692003061a"
format_stamp: "Formatted at 2026-06-20 19:12:35 on dist-test-slave-n3fx"
server_key: "5f518b7d8c2a8e6aae061b4da68b21af"
server_key_iv: "ef32bf5b29d8a758a29972e0a86100dc"
server_key_version: "encryptionkey@0"
I20260620 19:12:35.702205 23785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:35.717195 23785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:35.720357 23785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:35.720548 23785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:35.721128 23785 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:12:35.721940 23785 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:12:35.721992 23785 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:35.722036 23785 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:12:35.722064 23785 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:35.730295 23785 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.66:34855
I20260620 19:12:35.730338 23913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.66:34855 every 8 connection(s)
I20260620 19:12:35.731158 23785 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
I20260620 19:12:35.738631 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 23785
I20260620 19:12:35.738755 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-1/wal/instance
I20260620 19:12:35.739022 9837 external_mini_cluster.cc:1428] Setting key 757ba157a600a440842c31678ca10b85
Jun 20 19:12:35 dist-test-slave-n3fx krb5kdc[23544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982755, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.66@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:12:35.745183 23628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:35.733111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.66:54831 (local address 127.9.155.126:33551)
0620 19:12:35.733284 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:35.733288 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:35.734103 (+ 815us) server_negotiation.cc:407] Connection header received
0620 19:12:35.735439 (+ 1336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:35.735442 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:35.735500 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:35.735575 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:35.736807 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:35.737337 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:35.738518 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:35.738755 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:35.738852 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:35.741569 (+ 2717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:35.741572 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:35.741575 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:35.741600 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:35.743070 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:35.743584 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:35.743588 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:35.743590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:35.743670 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:35.743995 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:35.743999 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:35.744001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:35.744197 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:35.744317 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:35.744874 (+ 557us) server_negotiation.cc:299] Negotiation successful
0620 19:12:35.745012 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 19:12:35.746050 23916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:35.733473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:33551 (local address 127.9.155.66:54831)
0620 19:12:35.733929 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:35.733971 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:35.735220 (+ 1249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:35.735753 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:35.735759 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:35.736126 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:35.736656 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:35.736670 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:35.737456 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:35.737459 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:35.738395 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:35.738405 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:35.739019 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:35.739022 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:35.739195 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:35.739770 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:12:35.739790 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:35.741352 (+ 1562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:35.743187 (+ 1835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:35.743194 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:35.743212 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:35.743483 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:35.743755 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:35.743758 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:35.743760 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:35.743906 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:35.744324 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:35.744331 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:35.744603 (+ 272us) client_negotiation.cc:769] Sending connection context
0620 19:12:35.744814 (+ 211us) client_negotiation.cc:240] Negotiation successful
0620 19:12:35.745179 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":124,"threads_started":1}
I20260620 19:12:35.747430 23914 heartbeater.cc:347] Connected to a master server at 127.9.155.126:33551
I20260620 19:12:35.747690 23914 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.9.155.67@KRBTEST.COM; defaulting to no policy
I20260620 19:12:35.748239 23914 heartbeater.cc:511] Master 127.9.155.126:33551 requested a full tablet report, sending...
I20260620 19:12:35.749435 23588 ts_manager.cc:194] Registered new tserver with Master: f70693a176b0410cac470f692003061a (127.9.155.66:34855)
I20260620 19:12:35.750075 23588 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.155.66@KRBTEST.COM'} at 127.9.155.66:54831
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.155.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.155.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.67.keytab.
Entry for principal kudu/127.9.155.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.67.keytab.
I20260620 19:12:35.765198 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.67:0
--local_ip_for_outbound_sockets=127.9.155.67
--webserver_interface=127.9.155.67
--webserver_port=0
--tserver_master_addrs=127.9.155.126:33551
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.67.keytab
--principal=kudu/127.9.155.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:36051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:12:35.873874 23919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:35.874162 23919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:35.874205 23919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:35.878288 23919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:35.878361 23919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:35.878427 23919 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.67
I20260620 19:12:35.884246 23919 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:36051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.67.keytab
--principal=kudu/127.9.155.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.155.67
--webserver_port=0
--tserver_master_addrs=127.9.155.126:33551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23919
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.67
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:12:35.885367 23919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:35.886410 23919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:12:35.896586 23925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:35.896721 23919 server_base.cc:1034] running on GCE node
W20260620 19:12:35.896632 23927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:35.896639 23924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:35.897300 23919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:35.897950 23919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:35.899117 23919 hybrid_clock.cc:648] HybridClock initialized: now 1781982755899098 us; error 35 us; skew 500 ppm
Jun 20 19:12:35 dist-test-slave-n3fx krb5kdc[23544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982755, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:35.901989 23919 init.cc:377] Logged in from keytab as kudu/127.9.155.67@KRBTEST.COM (short username kudu)
I20260620 19:12:35.902897 23919 webserver.cc:460] Webserver started at http://127.9.155.67:45675/ using document root <none> and password file <none>
I20260620 19:12:35.903435 23919 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:35.903498 23919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:12:35.903683 23919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:12:35.905992 23919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-2/data/instance:
uuid: "037d4c71bd604f338452e010c6fd92f2"
format_stamp: "Formatted at 2026-06-20 19:12:35 on dist-test-slave-n3fx"
server_key: "e3e04ff741c4ff8b913153f78817607b"
server_key_iv: "bfb2376f4926a7c31508a84fa425eb44"
server_key_version: "encryptionkey@0"
I20260620 19:12:35.906641 23919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-2/wal/instance:
uuid: "037d4c71bd604f338452e010c6fd92f2"
format_stamp: "Formatted at 2026-06-20 19:12:35 on dist-test-slave-n3fx"
server_key: "e3e04ff741c4ff8b913153f78817607b"
server_key_iv: "bfb2376f4926a7c31508a84fa425eb44"
server_key_version: "encryptionkey@0"
I20260620 19:12:35.911000 23919 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 19:12:35.913837 23934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:35.915341 23919 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 19:12:35.915462 23919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-2/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-2/wal
uuid: "037d4c71bd604f338452e010c6fd92f2"
format_stamp: "Formatted at 2026-06-20 19:12:35 on dist-test-slave-n3fx"
server_key: "e3e04ff741c4ff8b913153f78817607b"
server_key_iv: "bfb2376f4926a7c31508a84fa425eb44"
server_key_version: "encryptionkey@0"
I20260620 19:12:35.915591 23919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:35.927268 23919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:35.930131 23919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:35.930300 23919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:35.930894 23919 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:12:35.931694 23919 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:12:35.931748 23919 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:35.931785 23919 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:12:35.931813 23919 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:35.940023 23919 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.67:44459
I20260620 19:12:35.940090 24047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.67:44459 every 8 connection(s)
I20260620 19:12:35.940964 23919 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
I20260620 19:12:35.941286 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 23919
I20260620 19:12:35.941411 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-2/wal/instance
I20260620 19:12:35.941792 9837 external_mini_cluster.cc:1428] Setting key c9ca65dd6beed5a1bb1b79dda23d4a51
Jun 20 19:12:35 dist-test-slave-n3fx krb5kdc[23544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982755, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.67@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:12:35.955541 23628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:35.942801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.67:47353 (local address 127.9.155.126:33551)
0620 19:12:35.943078 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:35.943082 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:35.943991 (+ 909us) server_negotiation.cc:407] Connection header received
0620 19:12:35.945388 (+ 1397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:35.945392 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:35.945440 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:35.945513 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:35.946890 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:35.947433 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:35.948808 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:35.949050 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:35.949131 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:35.951760 (+ 2629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:35.951763 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:35.951765 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:35.951792 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:35.953235 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:35.953729 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:35.953731 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:35.953732 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:35.953804 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:35.954104 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:35.954106 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:35.954107 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:35.954297 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:35.954374 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:35.955257 (+ 883us) server_negotiation.cc:299] Negotiation successful
0620 19:12:35.955388 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196}
I20260620 19:12:35.956483 24050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:35.943309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:33551 (local address 127.9.155.67:47353)
0620 19:12:35.943827 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:35.943887 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:35.945158 (+ 1271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:35.945733 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:35.945740 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:35.946129 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:35.946662 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:35.946677 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:35.947588 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:35.947591 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:35.948664 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:35.948675 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:35.949198 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:35.949201 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:35.949353 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:35.949955 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:12:35.949972 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:35.951562 (+ 1590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:35.953347 (+ 1785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:35.953352 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:35.953367 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:35.953641 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:35.953867 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:35.953870 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:35.953871 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:35.954019 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:35.954528 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:35.954534 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:35.954845 (+ 311us) client_negotiation.cc:769] Sending connection context
0620 19:12:35.955063 (+ 218us) client_negotiation.cc:240] Negotiation successful
0620 19:12:35.955480 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":132,"threads_started":1}
I20260620 19:12:35.957566 24048 heartbeater.cc:347] Connected to a master server at 127.9.155.126:33551
I20260620 19:12:35.957803 24048 heartbeater.cc:464] Registering TS with master...
I20260620 19:12:35.958274 24048 heartbeater.cc:511] Master 127.9.155.126:33551 requested a full tablet report, sending...
I20260620 19:12:35.959276 23588 ts_manager.cc:194] Registered new tserver with Master: 037d4c71bd604f338452e010c6fd92f2 (127.9.155.67:44459)
I20260620 19:12:35.959844 23588 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.155.67@KRBTEST.COM'} at 127.9.155.67:47353
I20260620 19:12:35.967970 9837 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:12:35.976963 23628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:35.969967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51834 (local address 127.9.155.126:33551)
0620 19:12:35.970084 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:35.970088 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:35.970276 (+ 188us) server_negotiation.cc:407] Connection header received
0620 19:12:35.970346 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:35.970348 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:35.970401 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:35.970481 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:35.971175 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:35.971687 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:35.972545 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:35.972777 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:35.972854 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:35.974051 (+ 1197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:35.974054 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:35.974056 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:35.974081 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:35.975305 (+ 1224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:35.975745 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:35.975747 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:35.975749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:35.975820 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:35.976091 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:35.976092 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:35.976094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:35.976341 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:35.976475 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:35.976735 (+ 260us) server_negotiation.cc:299] Negotiation successful
0620 19:12:35.976841 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 19:12:35.979960 23588 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51834:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:12:35.983184 23588 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:12:35.997879 24064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:35.993005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36424 (local address 127.9.155.66:34855)
0620 19:12:35.993738 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:35.993750 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:35.993771 (+ 21us) server_negotiation.cc:407] Connection header received
0620 19:12:35.993841 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:35.993845 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:35.994013 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:35.994160 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:12:35.994381 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:35.995339 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:35.996628 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:35.997201 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:35.997302 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:35.997463 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:35.997619 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 19:12:35.997717 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":122,"threads_started":1}
I20260620 19:12:35.997871 24060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:35.991894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.66:34855 (local address 127.0.0.1:36424)
0620 19:12:35.992674 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:35.992728 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:35.992843 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:35.994156 (+ 1313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:35.994159 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:35.994177 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:12:35.994273 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:35.994278 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:35.995460 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:35.995463 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:35.996510 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:35.996519 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:35.997317 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:35.997321 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:35.997435 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:35.997472 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 19:12:35.997580 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 19:12:35.997717 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":715,"mutex_wait_us":3,"spinlock_wait_cycles":7808,"thread_start_us":94,"threads_started":1}
I20260620 19:12:35.998099 24061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:35.991972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46032 (local address 127.9.155.67:44459)
0620 19:12:35.992593 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:35.992604 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:35.992973 (+ 369us) server_negotiation.cc:407] Connection header received
0620 19:12:35.993051 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:35.993057 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:35.993273 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:35.993430 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:12:35.993767 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:35.994941 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:35.996484 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:35.997323 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:35.997405 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:35.997539 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:35.997867 (+ 328us) server_negotiation.cc:299] Negotiation successful
0620 19:12:35.997958 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":141,"threads_started":1}
I20260620 19:12:35.998172 24063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:35.992516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33180 (local address 127.9.155.65:46309)
0620 19:12:35.993101 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:35.993115 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:35.993138 (+ 23us) server_negotiation.cc:407] Connection header received
0620 19:12:35.993211 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:35.993217 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:35.993384 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:35.993520 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:12:35.993875 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:35.994748 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:35.996482 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:35.997323 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:35.997420 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:35.997539 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:35.997954 (+ 415us) server_negotiation.cc:299] Negotiation successful
0620 19:12:35.998043 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":113,"threads_started":1}
I20260620 19:12:35.998613 24062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:35.991816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.67:44459 (local address 127.0.0.1:46032)
0620 19:12:35.992652 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:35.992703 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:35.992843 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:35.993436 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:35.993442 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:35.993464 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:12:35.993601 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:35.993609 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:35.995069 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:35.995073 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:35.996341 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:35.996352 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:35.997621 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:35.997624 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:35.997706 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:35.997752 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 19:12:35.998435 (+ 683us) client_negotiation.cc:240] Negotiation successful
0620 19:12:35.998509 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"mutex_wait_us":54,"spinlock_wait_cycles":6784,"thread_start_us":67,"threads_started":1}
I20260620 19:12:35.998921 24059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:35.992199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.65:46309 (local address 127.0.0.1:33180)
0620 19:12:35.992702 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:35.992714 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:35.992843 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:35.993658 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:35.993661 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:35.993677 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:12:35.993776 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:35.993781 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:35.994866 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:35.994870 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:35.996341 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:35.996352 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:35.997601 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:35.997605 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:35.997696 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:35.997742 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 19:12:35.997807 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 19:12:35.997884 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"mutex_wait_us":28,"spinlock_wait_cycles":3712,"thread_start_us":90,"threads_started":1}
I20260620 19:12:36.001169 23982 tablet_service.cc:1467] Processing CreateTablet for tablet 7f44ae14b316440b82fb0bb1e26140c3 (DEFAULT_TABLE table=test-table [id=1b33582e6b444252ab0f832c281e302a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:12:36.001178 23848 tablet_service.cc:1467] Processing CreateTablet for tablet 7f44ae14b316440b82fb0bb1e26140c3 (DEFAULT_TABLE table=test-table [id=1b33582e6b444252ab0f832c281e302a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:12:36.002185 23982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f44ae14b316440b82fb0bb1e26140c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:12:36.002221 23848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f44ae14b316440b82fb0bb1e26140c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:12:36.002130 23714 tablet_service.cc:1467] Processing CreateTablet for tablet 7f44ae14b316440b82fb0bb1e26140c3 (DEFAULT_TABLE table=test-table [id=1b33582e6b444252ab0f832c281e302a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:12:36.003190 23714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f44ae14b316440b82fb0bb1e26140c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:12:36.009284 24065 tablet_bootstrap.cc:492] T 7f44ae14b316440b82fb0bb1e26140c3 P 037d4c71bd604f338452e010c6fd92f2: Bootstrap starting.
I20260620 19:12:36.010161 24066 tablet_bootstrap.cc:492] T 7f44ae14b316440b82fb0bb1e26140c3 P f70693a176b0410cac470f692003061a: Bootstrap starting.
I20260620 19:12:36.010272 24067 tablet_bootstrap.cc:492] T 7f44ae14b316440b82fb0bb1e26140c3 P 2b13fa8f898b4c99b52c87c89708aa37: Bootstrap starting.
I20260620 19:12:36.012017 24067 tablet_bootstrap.cc:654] T 7f44ae14b316440b82fb0bb1e26140c3 P 2b13fa8f898b4c99b52c87c89708aa37: Neither blocks nor log segments found. Creating new log.
I20260620 19:12:36.012606 24065 tablet_bootstrap.cc:654] T 7f44ae14b316440b82fb0bb1e26140c3 P 037d4c71bd604f338452e010c6fd92f2: Neither blocks nor log segments found. Creating new log.
I20260620 19:12:36.012606 24066 tablet_bootstrap.cc:654] T 7f44ae14b316440b82fb0bb1e26140c3 P f70693a176b0410cac470f692003061a: Neither blocks nor log segments found. Creating new log.
I20260620 19:12:36.013212 24067 log.cc:824] T 7f44ae14b316440b82fb0bb1e26140c3 P 2b13fa8f898b4c99b52c87c89708aa37: Log is configured to *not* fsync() on all Append() calls
I20260620 19:12:36.013471 24066 log.cc:824] T 7f44ae14b316440b82fb0bb1e26140c3 P f70693a176b0410cac470f692003061a: Log is configured to *not* fsync() on all Append() calls
I20260620 19:12:36.013471 24065 log.cc:824] T 7f44ae14b316440b82fb0bb1e26140c3 P 037d4c71bd604f338452e010c6fd92f2: Log is configured to *not* fsync() on all Append() calls
I20260620 19:12:36.015308 24067 tablet_bootstrap.cc:492] T 7f44ae14b316440b82fb0bb1e26140c3 P 2b13fa8f898b4c99b52c87c89708aa37: No bootstrap required, opened a new log
I20260620 19:12:36.015479 24066 tablet_bootstrap.cc:492] T 7f44ae14b316440b82fb0bb1e26140c3 P f70693a176b0410cac470f692003061a: No bootstrap required, opened a new log
I20260620 19:12:36.015496 24067 ts_tablet_manager.cc:1397] T 7f44ae14b316440b82fb0bb1e26140c3 P 2b13fa8f898b4c99b52c87c89708aa37: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260620 19:12:36.015666 24065 tablet_bootstrap.cc:492] T 7f44ae14b316440b82fb0bb1e26140c3 P 037d4c71bd604f338452e010c6fd92f2: No bootstrap required, opened a new log
I20260620 19:12:36.015676 24066 ts_tablet_manager.cc:1397] T 7f44ae14b316440b82fb0bb1e26140c3 P f70693a176b0410cac470f692003061a: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260620 19:12:36.015833 24065 ts_tablet_manager.cc:1397] T 7f44ae14b316440b82fb0bb1e26140c3 P 037d4c71bd604f338452e010c6fd92f2: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260620 19:12:36.018581 24067 raft_consensus.cc:348] T 7f44ae14b316440b82fb0bb1e26140c3 P 2b13fa8f898b4c99b52c87c89708aa37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "037d4c71bd604f338452e010c6fd92f2" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 44459 } } peers { permanent_uuid: "f70693a176b0410cac470f692003061a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 34855 } } peers { permanent_uuid: "2b13fa8f898b4c99b52c87c89708aa37" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 46309 } }
I20260620 19:12:36.018823 24067 raft_consensus.cc:374] T 7f44ae14b316440b82fb0bb1e26140c3 P 2b13fa8f898b4c99b52c87c89708aa37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:12:36.018898 24067 raft_consensus.cc:729] T 7f44ae14b316440b82fb0bb1e26140c3 P 2b13fa8f898b4c99b52c87c89708aa37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b13fa8f898b4c99b52c87c89708aa37, State: Initialized, Role: FOLLOWER
I20260620 19:12:36.019343 24067 consensus_queue.cc:260] T 7f44ae14b316440b82fb0bb1e26140c3 P 2b13fa8f898b4c99b52c87c89708aa37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "037d4c71bd604f338452e010c6fd92f2" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 44459 } } peers { permanent_uuid: "f70693a176b0410cac470f692003061a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 34855 } } peers { permanent_uuid: "2b13fa8f898b4c99b52c87c89708aa37" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 46309 } }
I20260620 19:12:36.019582 24065 raft_consensus.cc:348] T 7f44ae14b316440b82fb0bb1e26140c3 P 037d4c71bd604f338452e010c6fd92f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "037d4c71bd604f338452e010c6fd92f2" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 44459 } } peers { permanent_uuid: "f70693a176b0410cac470f692003061a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 34855 } } peers { permanent_uuid: "2b13fa8f898b4c99b52c87c89708aa37" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 46309 } }
I20260620 19:12:36.019573 24066 raft_consensus.cc:348] T 7f44ae14b316440b82fb0bb1e26140c3 P f70693a176b0410cac470f692003061a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "037d4c71bd604f338452e010c6fd92f2" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 44459 } } peers { permanent_uuid: "f70693a176b0410cac470f692003061a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 34855 } } peers { permanent_uuid: "2b13fa8f898b4c99b52c87c89708aa37" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 46309 } }
I20260620 19:12:36.019840 24066 raft_consensus.cc:374] T 7f44ae14b316440b82fb0bb1e26140c3 P f70693a176b0410cac470f692003061a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:12:36.019840 24065 raft_consensus.cc:374] T 7f44ae14b316440b82fb0bb1e26140c3 P 037d4c71bd604f338452e010c6fd92f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:12:36.019901 24065 raft_consensus.cc:729] T 7f44ae14b316440b82fb0bb1e26140c3 P 037d4c71bd604f338452e010c6fd92f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 037d4c71bd604f338452e010c6fd92f2, State: Initialized, Role: FOLLOWER
I20260620 19:12:36.019901 24066 raft_consensus.cc:729] T 7f44ae14b316440b82fb0bb1e26140c3 P f70693a176b0410cac470f692003061a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f70693a176b0410cac470f692003061a, State: Initialized, Role: FOLLOWER
I20260620 19:12:36.020328 24067 ts_tablet_manager.cc:1428] T 7f44ae14b316440b82fb0bb1e26140c3 P 2b13fa8f898b4c99b52c87c89708aa37: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 19:12:36.020339 24066 consensus_queue.cc:260] T 7f44ae14b316440b82fb0bb1e26140c3 P f70693a176b0410cac470f692003061a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "037d4c71bd604f338452e010c6fd92f2" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 44459 } } peers { permanent_uuid: "f70693a176b0410cac470f692003061a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 34855 } } peers { permanent_uuid: "2b13fa8f898b4c99b52c87c89708aa37" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 46309 } }
I20260620 19:12:36.020339 24065 consensus_queue.cc:260] T 7f44ae14b316440b82fb0bb1e26140c3 P 037d4c71bd604f338452e010c6fd92f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "037d4c71bd604f338452e010c6fd92f2" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 44459 } } peers { permanent_uuid: "f70693a176b0410cac470f692003061a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 34855 } } peers { permanent_uuid: "2b13fa8f898b4c99b52c87c89708aa37" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 46309 } }
I20260620 19:12:36.020677 23780 heartbeater.cc:503] Master 127.9.155.126:33551 was elected leader, sending a full tablet report...
I20260620 19:12:36.021299 24065 ts_tablet_manager.cc:1428] T 7f44ae14b316440b82fb0bb1e26140c3 P 037d4c71bd604f338452e010c6fd92f2: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 19:12:36.021340 24066 ts_tablet_manager.cc:1428] T 7f44ae14b316440b82fb0bb1e26140c3 P f70693a176b0410cac470f692003061a: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260620 19:12:36.021442 23781 tablet.cc:2369] T 7f44ae14b316440b82fb0bb1e26140c3 P 2b13fa8f898b4c99b52c87c89708aa37: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:12:36.021653 23914 heartbeater.cc:503] Master 127.9.155.126:33551 was elected leader, sending a full tablet report...
I20260620 19:12:36.022670 24048 heartbeater.cc:503] Master 127.9.155.126:33551 was elected leader, sending a full tablet report...
W20260620 19:12:36.191987 24049 tablet.cc:2369] T 7f44ae14b316440b82fb0bb1e26140c3 P 037d4c71bd604f338452e010c6fd92f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 19:12:36.232167 23915 tablet.cc:2369] T 7f44ae14b316440b82fb0bb1e26140c3 P f70693a176b0410cac470f692003061a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:12:36.233070 24071 raft_consensus.cc:482] T 7f44ae14b316440b82fb0bb1e26140c3 P 2b13fa8f898b4c99b52c87c89708aa37 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:12:36.233249 24071 raft_consensus.cc:504] T 7f44ae14b316440b82fb0bb1e26140c3 P 2b13fa8f898b4c99b52c87c89708aa37 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "037d4c71bd604f338452e010c6fd92f2" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 44459 } } peers { permanent_uuid: "f70693a176b0410cac470f692003061a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 34855 } } peers { permanent_uuid: "2b13fa8f898b4c99b52c87c89708aa37" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 46309 } }
I20260620 19:12:36.234405 24071 leader_election.cc:288] T 7f44ae14b316440b82fb0bb1e26140c3 P 2b13fa8f898b4c99b52c87c89708aa37 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 037d4c71bd604f338452e010c6fd92f2 (127.9.155.67:44459), f70693a176b0410cac470f692003061a (127.9.155.66:34855)
I20260620 19:12:36.238875 24061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:36.234849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.65:51511 (local address 127.9.155.67:44459)
0620 19:12:36.234973 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:36.234978 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:36.235080 (+ 102us) server_negotiation.cc:407] Connection header received
0620 19:12:36.235309 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:36.235312 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:36.235355 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:36.235393 (+ 38us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:12:36.235870 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:36.236690 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:36.237882 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:36.238335 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:36.238455 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:36.238510 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:36.238683 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 19:12:36.238751 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 19:12:36.238899 24075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:36.234621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.67:44459 (local address 127.9.155.65:51511)
0620 19:12:36.234992 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:36.235005 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:36.235163 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:36.235519 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:36.235521 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:36.235554 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:12:36.235744 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:36.235749 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:36.236819 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:36.236822 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:36.237755 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:36.237766 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:36.238452 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:36.238455 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:36.238564 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:36.238591 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 19:12:36.238661 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 19:12:36.238738 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"mutex_wait_us":104,"spinlock_wait_cycles":768,"thread_start_us":112,"threads_started":1}
I20260620 19:12:36.239518 24064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:36.235114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.65:48223 (local address 127.9.155.66:34855)
0620 19:12:36.235238 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:36.235242 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:36.235257 (+ 15us) server_negotiation.cc:407] Connection header received
0620 19:12:36.235312 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:36.235315 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:36.235365 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:36.235448 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:12:36.236321 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:36.236856 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:36.238316 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:36.238847 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:36.238959 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:36.239007 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:36.239321 (+ 314us) server_negotiation.cc:299] Negotiation successful
0620 19:12:36.239403 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 19:12:36.239581 24002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f44ae14b316440b82fb0bb1e26140c3" candidate_uuid: "2b13fa8f898b4c99b52c87c89708aa37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "037d4c71bd604f338452e010c6fd92f2" is_pre_election: true
I20260620 19:12:36.239876 24002 raft_consensus.cc:2454] T 7f44ae14b316440b82fb0bb1e26140c3 P 037d4c71bd604f338452e010c6fd92f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b13fa8f898b4c99b52c87c89708aa37 in term 0.
I20260620 19:12:36.239915 24074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:36.234694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.66:34855 (local address 127.9.155.65:48223)
0620 19:12:36.235128 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:36.235140 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:36.235202 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:36.235478 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:36.235481 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:36.235499 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:12:36.235739 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:36.235747 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:36.236971 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:36.236975 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:36.238179 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:36.238189 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:36.239079 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:36.239083 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:36.239183 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:36.239206 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 19:12:36.239702 (+ 496us) client_negotiation.cc:240] Negotiation successful
0620 19:12:36.239776 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"mutex_wait_us":39,"spinlock_wait_cycles":33920,"thread_start_us":92,"threads_started":1}
I20260620 19:12:36.240370 23670 leader_election.cc:302] T 7f44ae14b316440b82fb0bb1e26140c3 P 2b13fa8f898b4c99b52c87c89708aa37 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 037d4c71bd604f338452e010c6fd92f2, 2b13fa8f898b4c99b52c87c89708aa37; no voters:
I20260620 19:12:36.240495 23868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f44ae14b316440b82fb0bb1e26140c3" candidate_uuid: "2b13fa8f898b4c99b52c87c89708aa37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f70693a176b0410cac470f692003061a" is_pre_election: true
I20260620 19:12:36.240643 24071 raft_consensus.cc:2781] T 7f44ae14b316440b82fb0bb1e26140c3 P 2b13fa8f898b4c99b52c87c89708aa37 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:12:36.240726 24071 raft_consensus.cc:482] T 7f44ae14b316440b82fb0bb1e26140c3 P 2b13fa8f898b4c99b52c87c89708aa37 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:12:36.240762 24071 raft_consensus.cc:3037] T 7f44ae14b316440b82fb0bb1e26140c3 P 2b13fa8f898b4c99b52c87c89708aa37 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:12:36.240762 23868 raft_consensus.cc:2454] T 7f44ae14b316440b82fb0bb1e26140c3 P f70693a176b0410cac470f692003061a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b13fa8f898b4c99b52c87c89708aa37 in term 0.
I20260620 19:12:36.241981 24071 raft_consensus.cc:504] T 7f44ae14b316440b82fb0bb1e26140c3 P 2b13fa8f898b4c99b52c87c89708aa37 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "037d4c71bd604f338452e010c6fd92f2" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 44459 } } peers { permanent_uuid: "f70693a176b0410cac470f692003061a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 34855 } } peers { permanent_uuid: "2b13fa8f898b4c99b52c87c89708aa37" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 46309 } }
I20260620 19:12:36.242328 24071 leader_election.cc:288] T 7f44ae14b316440b82fb0bb1e26140c3 P 2b13fa8f898b4c99b52c87c89708aa37 [CANDIDATE]: Term 1 election: Requested vote from peers 037d4c71bd604f338452e010c6fd92f2 (127.9.155.67:44459), f70693a176b0410cac470f692003061a (127.9.155.66:34855)
I20260620 19:12:36.242631 23868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f44ae14b316440b82fb0bb1e26140c3" candidate_uuid: "2b13fa8f898b4c99b52c87c89708aa37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f70693a176b0410cac470f692003061a"
I20260620 19:12:36.242631 24002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f44ae14b316440b82fb0bb1e26140c3" candidate_uuid: "2b13fa8f898b4c99b52c87c89708aa37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "037d4c71bd604f338452e010c6fd92f2"
I20260620 19:12:36.242745 23868 raft_consensus.cc:3037] T 7f44ae14b316440b82fb0bb1e26140c3 P f70693a176b0410cac470f692003061a [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:12:36.242758 24002 raft_consensus.cc:3037] T 7f44ae14b316440b82fb0bb1e26140c3 P 037d4c71bd604f338452e010c6fd92f2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:12:36.243974 23868 raft_consensus.cc:2454] T 7f44ae14b316440b82fb0bb1e26140c3 P f70693a176b0410cac470f692003061a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b13fa8f898b4c99b52c87c89708aa37 in term 1.
I20260620 19:12:36.243974 24002 raft_consensus.cc:2454] T 7f44ae14b316440b82fb0bb1e26140c3 P 037d4c71bd604f338452e010c6fd92f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b13fa8f898b4c99b52c87c89708aa37 in term 1.
I20260620 19:12:36.244300 23669 leader_election.cc:302] T 7f44ae14b316440b82fb0bb1e26140c3 P 2b13fa8f898b4c99b52c87c89708aa37 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b13fa8f898b4c99b52c87c89708aa37, f70693a176b0410cac470f692003061a; no voters:
I20260620 19:12:36.244521 24071 raft_consensus.cc:2781] T 7f44ae14b316440b82fb0bb1e26140c3 P 2b13fa8f898b4c99b52c87c89708aa37 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:12:36.244728 24071 raft_consensus.cc:686] T 7f44ae14b316440b82fb0bb1e26140c3 P 2b13fa8f898b4c99b52c87c89708aa37 [term 1 LEADER]: Becoming Leader. State: Replica: 2b13fa8f898b4c99b52c87c89708aa37, State: Running, Role: LEADER
I20260620 19:12:36.245056 24071 consensus_queue.cc:237] T 7f44ae14b316440b82fb0bb1e26140c3 P 2b13fa8f898b4c99b52c87c89708aa37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "037d4c71bd604f338452e010c6fd92f2" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 44459 } } peers { permanent_uuid: "f70693a176b0410cac470f692003061a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 34855 } } peers { permanent_uuid: "2b13fa8f898b4c99b52c87c89708aa37" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 46309 } }
I20260620 19:12:36.248458 23589 catalog_manager.cc:5495] T 7f44ae14b316440b82fb0bb1e26140c3 P 2b13fa8f898b4c99b52c87c89708aa37 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2b13fa8f898b4c99b52c87c89708aa37 (127.9.155.65). New cstate: current_term: 1 leader_uuid: "2b13fa8f898b4c99b52c87c89708aa37" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "037d4c71bd604f338452e010c6fd92f2" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 44459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f70693a176b0410cac470f692003061a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 34855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b13fa8f898b4c99b52c87c89708aa37" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 46309 } health_report { overall_health: HEALTHY } } }
I20260620 19:12:36.291975 24063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:36.288530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33188 (local address 127.9.155.65:46309)
0620 19:12:36.288685 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:36.288690 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:36.288704 (+ 14us) server_negotiation.cc:407] Connection header received
0620 19:12:36.288783 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:36.288786 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:36.288841 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:36.288928 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:12:36.289136 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:36.289627 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:36.290460 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:36.290679 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:36.290764 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:36.291076 (+ 312us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:12:36.291505 (+ 429us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:12:36.291592 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:36.291764 (+ 172us) server_negotiation.cc:299] Negotiation successful
0620 19:12:36.291833 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 19:12:36.297610 24002 raft_consensus.cc:1261] T 7f44ae14b316440b82fb0bb1e26140c3 P 037d4c71bd604f338452e010c6fd92f2 [term 1 FOLLOWER]: Refusing update from remote peer 2b13fa8f898b4c99b52c87c89708aa37: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:12:36.297652 23868 raft_consensus.cc:1261] T 7f44ae14b316440b82fb0bb1e26140c3 P f70693a176b0410cac470f692003061a [term 1 FOLLOWER]: Refusing update from remote peer 2b13fa8f898b4c99b52c87c89708aa37: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:12:36.298333 24076 consensus_queue.cc:1035] T 7f44ae14b316440b82fb0bb1e26140c3 P 2b13fa8f898b4c99b52c87c89708aa37 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f70693a176b0410cac470f692003061a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 34855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:12:36.298516 24071 consensus_queue.cc:1035] T 7f44ae14b316440b82fb0bb1e26140c3 P 2b13fa8f898b4c99b52c87c89708aa37 [LEADER]: Connected to new peer: Peer: permanent_uuid: "037d4c71bd604f338452e010c6fd92f2" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 44459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:12:36.306906 24080 mvcc.cc:205] Tried to move back new op lower bound from 7299001369788231680 to 7299001369582526464. Current Snapshot: MvccSnapshot[applied={T|T < 7299001369788231680}]
I20260620 19:12:36.313643 23589 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51834:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:12:36.313853 23589 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51834:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:12:36.316489 23589 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b34ec788c4684b4bb78bf6fed9657a8c: Sending DeleteTablet for 3 replicas of tablet 7f44ae14b316440b82fb0bb1e26140c3
I20260620 19:12:36.317327 23714 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f44ae14b316440b82fb0bb1e26140c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:12:36 UTC) from {username='kudu', principal='kudu/127.9.155.126@KRBTEST.COM'} at 127.0.0.1:33180
I20260620 19:12:36.317878 24088 tablet_replica.cc:330] T 7f44ae14b316440b82fb0bb1e26140c3 P 2b13fa8f898b4c99b52c87c89708aa37: stopping tablet replica
I20260620 19:12:36.318135 23848 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f44ae14b316440b82fb0bb1e26140c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:12:36 UTC) from {username='kudu', principal='kudu/127.9.155.126@KRBTEST.COM'} at 127.0.0.1:36424
I20260620 19:12:36.318385 24088 raft_consensus.cc:2229] T 7f44ae14b316440b82fb0bb1e26140c3 P 2b13fa8f898b4c99b52c87c89708aa37 [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:12:36.318931 24088 raft_consensus.cc:2258] T 7f44ae14b316440b82fb0bb1e26140c3 P 2b13fa8f898b4c99b52c87c89708aa37 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:12:36.318977 24089 tablet_replica.cc:330] T 7f44ae14b316440b82fb0bb1e26140c3 P f70693a176b0410cac470f692003061a: stopping tablet replica
I20260620 19:12:36.319249 24089 raft_consensus.cc:2229] T 7f44ae14b316440b82fb0bb1e26140c3 P f70693a176b0410cac470f692003061a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:12:36.319538 24089 raft_consensus.cc:2258] T 7f44ae14b316440b82fb0bb1e26140c3 P f70693a176b0410cac470f692003061a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:12:36.321071 24089 ts_tablet_manager.cc:1905] T 7f44ae14b316440b82fb0bb1e26140c3 P f70693a176b0410cac470f692003061a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:12:36.321583 23982 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f44ae14b316440b82fb0bb1e26140c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:12:36 UTC) from {username='kudu', principal='kudu/127.9.155.126@KRBTEST.COM'} at 127.0.0.1:46032
I20260620 19:12:36.321731 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 23651
I20260620 19:12:36.322824 24090 tablet_replica.cc:330] T 7f44ae14b316440b82fb0bb1e26140c3 P 037d4c71bd604f338452e010c6fd92f2: stopping tablet replica
I20260620 19:12:36.323102 24090 raft_consensus.cc:2229] T 7f44ae14b316440b82fb0bb1e26140c3 P 037d4c71bd604f338452e010c6fd92f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:12:36.323397 24090 raft_consensus.cc:2258] T 7f44ae14b316440b82fb0bb1e26140c3 P 037d4c71bd604f338452e010c6fd92f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:12:36.323784 24089 ts_tablet_manager.cc:1918] T 7f44ae14b316440b82fb0bb1e26140c3 P f70693a176b0410cac470f692003061a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:12:36.323877 24089 log.cc:1196] T 7f44ae14b316440b82fb0bb1e26140c3 P f70693a176b0410cac470f692003061a: Deleting WAL directory at /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-1/wal/wals/7f44ae14b316440b82fb0bb1e26140c3
I20260620 19:12:36.324110 24089 ts_tablet_manager.cc:1939] T 7f44ae14b316440b82fb0bb1e26140c3 P f70693a176b0410cac470f692003061a: Deleting consensus metadata
I20260620 19:12:36.324972 24090 ts_tablet_manager.cc:1905] T 7f44ae14b316440b82fb0bb1e26140c3 P 037d4c71bd604f338452e010c6fd92f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:12:36.325213 23575 catalog_manager.cc:4841] TS f70693a176b0410cac470f692003061a (127.9.155.66:34855): tablet 7f44ae14b316440b82fb0bb1e26140c3 (table test-table [id=1b33582e6b444252ab0f832c281e302a]) successfully deleted
I20260620 19:12:36.327574 24090 ts_tablet_manager.cc:1918] T 7f44ae14b316440b82fb0bb1e26140c3 P 037d4c71bd604f338452e010c6fd92f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:12:36.327661 24090 log.cc:1196] T 7f44ae14b316440b82fb0bb1e26140c3 P 037d4c71bd604f338452e010c6fd92f2: Deleting WAL directory at /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-2/wal/wals/7f44ae14b316440b82fb0bb1e26140c3
I20260620 19:12:36.327961 24090 ts_tablet_manager.cc:1939] T 7f44ae14b316440b82fb0bb1e26140c3 P 037d4c71bd604f338452e010c6fd92f2: Deleting consensus metadata
W20260620 19:12:36.329150 23574 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.155.65:46309 (error 108)
I20260620 19:12:36.329308 23576 catalog_manager.cc:4841] TS 037d4c71bd604f338452e010c6fd92f2 (127.9.155.67:44459): tablet 7f44ae14b316440b82fb0bb1e26140c3 (table test-table [id=1b33582e6b444252ab0f832c281e302a]) successfully deleted
I20260620 19:12:36.329560 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 23785
I20260620 19:12:36.333356 24059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:36.333038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.65:46309 (local address 127.0.0.1:33194)
0620 19:12:36.333160 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:36.333261 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.65:46309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 19:12:36.333587 23574 catalog_manager.cc:4584] TS 2b13fa8f898b4c99b52c87c89708aa37 (127.9.155.65:46309): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7f44ae14b316440b82fb0bb1e26140c3: Network error: Client connection negotiation failed: client connection to 127.9.155.65:46309: connect: Connection refused (error 111)
I20260620 19:12:36.334379 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 23919
I20260620 19:12:36.339020 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 23557
I20260620 19:12:36.344100 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:33551
--webserver_interface=127.9.155.126
--webserver_port=40697
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab
--principal=kudu/127.9.155.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:36051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.155.126:33551
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:12:36.453799 24092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:36.454063 24092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:36.454103 24092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:36.458161 24092 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:12:36.458240 24092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:36.458259 24092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:36.458276 24092 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:12:36.458290 24092 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:12:36.463652 24092 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:36051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.9.155.126:33551
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab
--principal=kudu/127.9.155.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:33551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.155.126
--webserver_port=40697
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24092
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:12:36.464753 24092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:36.465746 24092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:12:36.478119 24098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:36.478119 24101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:36.478119 24099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:36.478380 24092 server_base.cc:1034] running on GCE node
I20260620 19:12:36.479070 24092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:36.480243 24092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:36.481411 24092 hybrid_clock.cc:648] HybridClock initialized: now 1781982756481390 us; error 33 us; skew 500 ppm
Jun 20 19:12:36 dist-test-slave-n3fx krb5kdc[23544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982756, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:36.484032 24092 init.cc:377] Logged in from keytab as kudu/127.9.155.126@KRBTEST.COM (short username kudu)
I20260620 19:12:36.484988 24092 webserver.cc:460] Webserver started at http://127.9.155.126:40697/ using document root <none> and password file <none>
I20260620 19:12:36.485532 24092 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:36.485611 24092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:12:36.489526 24092 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 19:12:36.491641 24108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:36.492889 24092 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:12:36.493000 24092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/master-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/master-0/wal
uuid: "b34ec788c4684b4bb78bf6fed9657a8c"
format_stamp: "Formatted at 2026-06-20 19:12:35 on dist-test-slave-n3fx"
server_key: "4e368b3c54979a8b4297115bc51f9dcd"
server_key_iv: "cbad7a6c1833629c473316f986ab7d22"
server_key_version: "encryptionkey@0"
I20260620 19:12:36.493566 24092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:36.511907 24092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:36.515303 24092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:36.515486 24092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:36.522612 24092 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.126:33551
I20260620 19:12:36.522969 24160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.126:33551 every 8 connection(s)
I20260620 19:12:36.523736 24092 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
I20260620 19:12:36.528407 24161 sys_catalog.cc:263] Verifying existing consensus state
I20260620 19:12:36.530107 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 24092
I20260620 19:12:36.530280 24161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b34ec788c4684b4bb78bf6fed9657a8c: Bootstrap starting.
I20260620 19:12:36.530532 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.65:46309
--local_ip_for_outbound_sockets=127.9.155.65
--tserver_master_addrs=127.9.155.126:33551
--webserver_port=40617
--webserver_interface=127.9.155.65
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.65.keytab
--principal=kudu/127.9.155.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:36051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:12:36.547241 24161 log.cc:824] T 00000000000000000000000000000000 P b34ec788c4684b4bb78bf6fed9657a8c: Log is configured to *not* fsync() on all Append() calls
I20260620 19:12:36.556193 24161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b34ec788c4684b4bb78bf6fed9657a8c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:12:36.556623 24161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b34ec788c4684b4bb78bf6fed9657a8c: Bootstrap complete.
I20260620 19:12:36.560611 24161 raft_consensus.cc:348] T 00000000000000000000000000000000 P b34ec788c4684b4bb78bf6fed9657a8c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b34ec788c4684b4bb78bf6fed9657a8c" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 33551 } }
I20260620 19:12:36.561012 24161 raft_consensus.cc:729] T 00000000000000000000000000000000 P b34ec788c4684b4bb78bf6fed9657a8c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b34ec788c4684b4bb78bf6fed9657a8c, State: Initialized, Role: FOLLOWER
I20260620 19:12:36.561517 24161 consensus_queue.cc:260] T 00000000000000000000000000000000 P b34ec788c4684b4bb78bf6fed9657a8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b34ec788c4684b4bb78bf6fed9657a8c" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 33551 } }
I20260620 19:12:36.561637 24161 raft_consensus.cc:388] T 00000000000000000000000000000000 P b34ec788c4684b4bb78bf6fed9657a8c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:12:36.561707 24161 raft_consensus.cc:482] T 00000000000000000000000000000000 P b34ec788c4684b4bb78bf6fed9657a8c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:12:36.561794 24161 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b34ec788c4684b4bb78bf6fed9657a8c [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:12:36.563665 24161 raft_consensus.cc:504] T 00000000000000000000000000000000 P b34ec788c4684b4bb78bf6fed9657a8c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b34ec788c4684b4bb78bf6fed9657a8c" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 33551 } }
I20260620 19:12:36.563979 24161 leader_election.cc:302] T 00000000000000000000000000000000 P b34ec788c4684b4bb78bf6fed9657a8c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b34ec788c4684b4bb78bf6fed9657a8c; no voters:
I20260620 19:12:36.564338 24161 leader_election.cc:288] T 00000000000000000000000000000000 P b34ec788c4684b4bb78bf6fed9657a8c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 19:12:36.564443 24165 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b34ec788c4684b4bb78bf6fed9657a8c [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:12:36.564684 24165 raft_consensus.cc:686] T 00000000000000000000000000000000 P b34ec788c4684b4bb78bf6fed9657a8c [term 2 LEADER]: Becoming Leader. State: Replica: b34ec788c4684b4bb78bf6fed9657a8c, State: Running, Role: LEADER
I20260620 19:12:36.564975 24165 consensus_queue.cc:237] T 00000000000000000000000000000000 P b34ec788c4684b4bb78bf6fed9657a8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b34ec788c4684b4bb78bf6fed9657a8c" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 33551 } }
I20260620 19:12:36.565500 24161 sys_catalog.cc:564] T 00000000000000000000000000000000 P b34ec788c4684b4bb78bf6fed9657a8c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:12:36.566279 24166 sys_catalog.cc:455] T 00000000000000000000000000000000 P b34ec788c4684b4bb78bf6fed9657a8c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b34ec788c4684b4bb78bf6fed9657a8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b34ec788c4684b4bb78bf6fed9657a8c" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 33551 } } }
I20260620 19:12:36.566402 24166 sys_catalog.cc:458] T 00000000000000000000000000000000 P b34ec788c4684b4bb78bf6fed9657a8c [sys.catalog]: This master's current role is: LEADER
I20260620 19:12:36.566704 24167 sys_catalog.cc:455] T 00000000000000000000000000000000 P b34ec788c4684b4bb78bf6fed9657a8c [sys.catalog]: SysCatalogTable state changed. Reason: New leader b34ec788c4684b4bb78bf6fed9657a8c. Latest consensus state: current_term: 2 leader_uuid: "b34ec788c4684b4bb78bf6fed9657a8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b34ec788c4684b4bb78bf6fed9657a8c" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 33551 } } }
I20260620 19:12:36.566784 24167 sys_catalog.cc:458] T 00000000000000000000000000000000 P b34ec788c4684b4bb78bf6fed9657a8c [sys.catalog]: This master's current role is: LEADER
I20260620 19:12:36.567435 24174 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:12:36.572504 24174 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:12:36.573185 24174 catalog_manager.cc:1259] Loaded cluster ID: d9a35173f7a344e9b05224cb361a8f31
I20260620 19:12:36.573251 24174 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:12:36.574666 24174 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260620 19:12:36.579452 24174 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260620 19:12:36.585213 24174 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:12:36.586009 24174 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b34ec788c4684b4bb78bf6fed9657a8c: Loaded TSK: 0
I20260620 19:12:36.587235 24174 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 19:12:36.642215 24163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:36.642499 24163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:36.642552 24163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:36.646934 24163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:36.647010 24163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:36.647073 24163 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.65
I20260620 19:12:36.652554 24163 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:36051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.65.keytab
--principal=kudu/127.9.155.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.65:46309
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.155.65
--webserver_port=40617
--tserver_master_addrs=127.9.155.126:33551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24163
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.65
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:12:36.653568 24163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:36.654541 24163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:12:36.664125 24188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:36.664093 24189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:36.664139 24191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:36.664685 24163 server_base.cc:1034] running on GCE node
I20260620 19:12:36.665089 24163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:36.665823 24163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:36.666988 24163 hybrid_clock.cc:648] HybridClock initialized: now 1781982756666964 us; error 42 us; skew 500 ppm
Jun 20 19:12:36 dist-test-slave-n3fx krb5kdc[23544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982756, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:36.669783 24163 init.cc:377] Logged in from keytab as kudu/127.9.155.65@KRBTEST.COM (short username kudu)
I20260620 19:12:36.670703 24163 webserver.cc:460] Webserver started at http://127.9.155.65:40617/ using document root <none> and password file <none>
I20260620 19:12:36.671238 24163 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:36.671309 24163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:12:36.675374 24163 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 19:12:36.678061 24198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:36.679524 24163 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 19:12:36.679656 24163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-0/wal
uuid: "2b13fa8f898b4c99b52c87c89708aa37"
format_stamp: "Formatted at 2026-06-20 19:12:35 on dist-test-slave-n3fx"
server_key: "6a53c726ff4021dba8d7ac3f9460e48a"
server_key_iv: "ab9796f5b048326491fe0833c8761fd0"
server_key_version: "encryptionkey@0"
I20260620 19:12:36.680404 24163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:36.705061 24163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:36.708952 24163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:36.709146 24163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:36.709690 24163 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:12:36.710713 24205 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 19:12:36.715624 24163 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 19:12:36.715695 24163 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260620 19:12:36.715775 24163 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 19:12:36.717439 24163 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 19:12:36.717484 24163 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260620 19:12:36.717777 24205 tablet_bootstrap.cc:492] T 7f44ae14b316440b82fb0bb1e26140c3 P 2b13fa8f898b4c99b52c87c89708aa37: Bootstrap starting.
I20260620 19:12:36.726205 24163 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.65:46309
I20260620 19:12:36.726261 24312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.65:46309 every 8 connection(s)
I20260620 19:12:36.727093 24163 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
I20260620 19:12:36.736750 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 24163
I20260620 19:12:36.736869 24205 log.cc:824] T 7f44ae14b316440b82fb0bb1e26140c3 P 2b13fa8f898b4c99b52c87c89708aa37: Log is configured to *not* fsync() on all Append() calls
I20260620 19:12:36.737195 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.66:34855
--local_ip_for_outbound_sockets=127.9.155.66
--tserver_master_addrs=127.9.155.126:33551
--webserver_port=33871
--webserver_interface=127.9.155.66
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.66.keytab
--principal=kudu/127.9.155.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:36051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 19:12:36 dist-test-slave-n3fx krb5kdc[23544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982756, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.65@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:12:36.743870 24315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:36.728965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.65:41797 (local address 127.9.155.126:33551)
0620 19:12:36.729365 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:36.729385 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:36.729769 (+ 384us) server_negotiation.cc:407] Connection header received
0620 19:12:36.731092 (+ 1323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:36.731105 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:36.731385 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:36.731692 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:36.733117 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:36.734099 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:36.735773 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:36.736205 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:36.736350 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:36.739214 (+ 2864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:36.739220 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:36.739236 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:36.739265 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:36.740936 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:36.741475 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:36.741479 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:36.741483 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:36.741587 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:36.741946 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:36.741948 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:36.741950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:36.742274 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:36.742371 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:36.742948 (+ 577us) server_negotiation.cc:299] Negotiation successful
0620 19:12:36.743192 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":129,"threads_started":1}
I20260620 19:12:36.743983 24316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:36.728943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:33551 (local address 127.9.155.65:41797)
0620 19:12:36.729595 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:36.729650 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:36.730689 (+ 1039us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:36.731790 (+ 1101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:36.731800 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:36.732352 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:36.732958 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:36.732977 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:36.734266 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:36.734270 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:36.735663 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:36.735673 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:36.736349 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:36.736352 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:36.736537 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:36.737177 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:12:36.737198 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:36.738995 (+ 1797us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:36.741051 (+ 2056us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:36.741056 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:36.741067 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:36.741383 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:36.741706 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:36.741709 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:36.741710 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:36.741864 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:36.742381 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:36.742386 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:36.742646 (+ 260us) client_negotiation.cc:769] Sending connection context
0620 19:12:36.742836 (+ 190us) client_negotiation.cc:240] Negotiation successful
0620 19:12:36.743147 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":127,"threads_started":1}
I20260620 19:12:36.744935 24205 tablet_bootstrap.cc:492] T 7f44ae14b316440b82fb0bb1e26140c3 P 2b13fa8f898b4c99b52c87c89708aa37: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:12:36.745349 24205 tablet_bootstrap.cc:492] T 7f44ae14b316440b82fb0bb1e26140c3 P 2b13fa8f898b4c99b52c87c89708aa37: Bootstrap complete.
I20260620 19:12:36.746335 24205 ts_tablet_manager.cc:1397] T 7f44ae14b316440b82fb0bb1e26140c3 P 2b13fa8f898b4c99b52c87c89708aa37: Time spent bootstrapping tablet: real 0.029s user 0.026s sys 0.000s
I20260620 19:12:36.746786 24313 heartbeater.cc:347] Connected to a master server at 127.9.155.126:33551
I20260620 19:12:36.747008 24313 heartbeater.cc:464] Registering TS with master...
I20260620 19:12:36.747540 24313 heartbeater.cc:511] Master 127.9.155.126:33551 requested a full tablet report, sending...
I20260620 19:12:36.749158 24205 raft_consensus.cc:348] T 7f44ae14b316440b82fb0bb1e26140c3 P 2b13fa8f898b4c99b52c87c89708aa37 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "037d4c71bd604f338452e010c6fd92f2" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 44459 } } peers { permanent_uuid: "f70693a176b0410cac470f692003061a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 34855 } } peers { permanent_uuid: "2b13fa8f898b4c99b52c87c89708aa37" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 46309 } }
I20260620 19:12:36.749570 24205 raft_consensus.cc:729] T 7f44ae14b316440b82fb0bb1e26140c3 P 2b13fa8f898b4c99b52c87c89708aa37 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b13fa8f898b4c99b52c87c89708aa37, State: Initialized, Role: FOLLOWER
I20260620 19:12:36.750005 24205 consensus_queue.cc:260] T 7f44ae14b316440b82fb0bb1e26140c3 P 2b13fa8f898b4c99b52c87c89708aa37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "037d4c71bd604f338452e010c6fd92f2" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 44459 } } peers { permanent_uuid: "f70693a176b0410cac470f692003061a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 34855 } } peers { permanent_uuid: "2b13fa8f898b4c99b52c87c89708aa37" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 46309 } }
I20260620 19:12:36.750389 24125 ts_manager.cc:194] Registered new tserver with Master: 2b13fa8f898b4c99b52c87c89708aa37 (127.9.155.65:46309)
I20260620 19:12:36.750921 24205 ts_tablet_manager.cc:1428] T 7f44ae14b316440b82fb0bb1e26140c3 P 2b13fa8f898b4c99b52c87c89708aa37: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 19:12:36.754707 24125 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.155.65@KRBTEST.COM'} at 127.9.155.65:41797
I20260620 19:12:36.757500 24313 heartbeater.cc:503] Master 127.9.155.126:33551 was elected leader, sending a full tablet report...
Jun 20 19:12:36 dist-test-slave-n3fx krb5kdc[23544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982756, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.126@KRBTEST.COM for kudu/127.9.155.65@KRBTEST.COM
I20260620 19:12:36.764147 24320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:36.754006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.65:46309 (local address 127.0.0.1:33198)
0620 19:12:36.754674 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:36.754708 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:36.754855 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:36.755271 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:36.755277 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:36.755672 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:36.755793 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:36.755801 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:36.757272 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:36.757276 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:36.758100 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:36.758109 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:36.758640 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:36.758644 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:36.758762 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:36.759297 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:12:36.759311 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:36.760880 (+ 1569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:36.762494 (+ 1614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:36.762503 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:36.762509 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:36.762728 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:36.762988 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:36.762991 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:36.762993 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:36.763069 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:36.763489 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:36.763495 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:36.763612 (+ 117us) client_negotiation.cc:769] Sending connection context
0620 19:12:36.763762 (+ 150us) client_negotiation.cc:240] Negotiation successful
0620 19:12:36.763967 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":93,"threads_started":1}
I20260620 19:12:36.764287 24321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:36.754084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33198 (local address 127.9.155.65:46309)
0620 19:12:36.754438 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:36.754450 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:36.754887 (+ 437us) server_negotiation.cc:407] Connection header received
0620 19:12:36.754983 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:36.754987 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:36.755142 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:36.755265 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:36.755907 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:36.757134 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:36.758195 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:36.758534 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:36.758625 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:36.761000 (+ 2375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:36.761008 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:36.761018 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:36.761039 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:36.762395 (+ 1356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:36.762827 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:36.762832 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:36.762833 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:36.762908 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:36.763144 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:36.763147 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:36.763149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:36.763384 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:36.763498 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:36.763913 (+ 415us) server_negotiation.cc:299] Negotiation successful
0620 19:12:36.764074 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":261,"thread_start_us":77,"threads_started":1}
I20260620 19:12:36.765609 24247 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f44ae14b316440b82fb0bb1e26140c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:12:36 UTC) from {username='kudu', principal='kudu/127.9.155.126@KRBTEST.COM'} at 127.0.0.1:33198
I20260620 19:12:36.766501 24322 tablet_replica.cc:330] stopping tablet replica
I20260620 19:12:36.766939 24322 raft_consensus.cc:2229] T 7f44ae14b316440b82fb0bb1e26140c3 P 2b13fa8f898b4c99b52c87c89708aa37 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:12:36.767202 24322 raft_consensus.cc:2258] T 7f44ae14b316440b82fb0bb1e26140c3 P 2b13fa8f898b4c99b52c87c89708aa37 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:12:36.768316 24322 ts_tablet_manager.cc:1905] T 7f44ae14b316440b82fb0bb1e26140c3 P 2b13fa8f898b4c99b52c87c89708aa37: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:12:36.773066 24322 ts_tablet_manager.cc:1918] T 7f44ae14b316440b82fb0bb1e26140c3 P 2b13fa8f898b4c99b52c87c89708aa37: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:12:36.773190 24322 log.cc:1196] T 7f44ae14b316440b82fb0bb1e26140c3 P 2b13fa8f898b4c99b52c87c89708aa37: Deleting WAL directory at /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-0/wal/wals/7f44ae14b316440b82fb0bb1e26140c3
I20260620 19:12:36.773473 24322 ts_tablet_manager.cc:1939] T 7f44ae14b316440b82fb0bb1e26140c3 P 2b13fa8f898b4c99b52c87c89708aa37: Deleting consensus metadata
I20260620 19:12:36.775612 24110 catalog_manager.cc:4841] TS 2b13fa8f898b4c99b52c87c89708aa37 (127.9.155.65:46309): tablet 7f44ae14b316440b82fb0bb1e26140c3 (table test-table [id=1b33582e6b444252ab0f832c281e302a]) successfully deleted
W20260620 19:12:36.859818 24318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:36.860105 24318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:36.860148 24318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:36.864284 24318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:36.864352 24318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:36.864420 24318 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.66
I20260620 19:12:36.869983 24318 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:36051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.66.keytab
--principal=kudu/127.9.155.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.66:34855
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.155.66
--webserver_port=33871
--tserver_master_addrs=127.9.155.126:33551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24318
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.66
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:12:36.871062 24318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:36.872061 24318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:12:36.881448 24327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:36.881547 24318 server_base.cc:1034] running on GCE node
W20260620 19:12:36.881476 24328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:36.881457 24330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:36.882133 24318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:36.882761 24318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:36.883937 24318 hybrid_clock.cc:648] HybridClock initialized: now 1781982756883918 us; error 35 us; skew 500 ppm
Jun 20 19:12:36 dist-test-slave-n3fx krb5kdc[23544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982756, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:36.886636 24318 init.cc:377] Logged in from keytab as kudu/127.9.155.66@KRBTEST.COM (short username kudu)
I20260620 19:12:36.887584 24318 webserver.cc:460] Webserver started at http://127.9.155.66:33871/ using document root <none> and password file <none>
I20260620 19:12:36.888130 24318 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:36.888242 24318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:12:36.892277 24318 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 19:12:36.894297 24337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:36.895517 24318 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 19:12:36.895643 24318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-1/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-1/wal
uuid: "f70693a176b0410cac470f692003061a"
format_stamp: "Formatted at 2026-06-20 19:12:35 on dist-test-slave-n3fx"
server_key: "5f518b7d8c2a8e6aae061b4da68b21af"
server_key_iv: "ef32bf5b29d8a758a29972e0a86100dc"
server_key_version: "encryptionkey@0"
I20260620 19:12:36.896178 24318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:36.913419 24318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:36.916697 24318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:36.916880 24318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:36.917464 24318 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:12:36.918303 24318 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:12:36.918349 24318 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:36.918401 24318 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:12:36.918428 24318 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:36.926705 24318 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.66:34855
I20260620 19:12:36.926728 24450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.66:34855 every 8 connection(s)
I20260620 19:12:36.927790 24318 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
I20260620 19:12:36.928795 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 24318
I20260620 19:12:36.929261 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.67:44459
--local_ip_for_outbound_sockets=127.9.155.67
--tserver_master_addrs=127.9.155.126:33551
--webserver_port=45675
--webserver_interface=127.9.155.67
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.67.keytab
--principal=kudu/127.9.155.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:36051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 19:12:36 dist-test-slave-n3fx krb5kdc[23544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982756, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.66@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:12:36.941545 24315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:36.929726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.66:52497 (local address 127.9.155.126:33551)
0620 19:12:36.929872 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:36.929877 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:36.930734 (+ 857us) server_negotiation.cc:407] Connection header received
0620 19:12:36.931971 (+ 1237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:36.931975 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:36.932036 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:36.932119 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:36.933350 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:36.933873 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:36.935013 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:36.935260 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:36.935353 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:36.938080 (+ 2727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:36.938083 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:36.938085 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:36.938113 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:36.939467 (+ 1354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:36.940036 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:36.940039 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:36.940040 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:36.940119 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:36.940485 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:36.940488 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:36.940489 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:36.940669 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:36.940744 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:36.941306 (+ 562us) server_negotiation.cc:299] Negotiation successful
0620 19:12:36.941419 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 19:12:36.942561 24454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:36.930098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:33551 (local address 127.9.155.66:52497)
0620 19:12:36.930586 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:36.930636 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:36.931772 (+ 1136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:36.932320 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:36.932327 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:36.932728 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:36.933206 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:36.933219 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:36.933990 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:36.933993 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:36.934894 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:36.934904 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:36.935405 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:36.935409 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:36.935581 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:36.936245 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:12:36.936270 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:36.937883 (+ 1613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:36.939575 (+ 1692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:36.939581 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:36.939597 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:36.939932 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:36.940192 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:36.940195 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:36.940197 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:36.940390 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:36.940755 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:36.940761 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:36.941065 (+ 304us) client_negotiation.cc:769] Sending connection context
0620 19:12:36.941260 (+ 195us) client_negotiation.cc:240] Negotiation successful
0620 19:12:36.941586 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":136,"threads_started":1}
I20260620 19:12:36.943717 24451 heartbeater.cc:347] Connected to a master server at 127.9.155.126:33551
I20260620 19:12:36.943950 24451 heartbeater.cc:464] Registering TS with master...
I20260620 19:12:36.944456 24451 heartbeater.cc:511] Master 127.9.155.126:33551 requested a full tablet report, sending...
I20260620 19:12:36.945428 24125 ts_manager.cc:194] Registered new tserver with Master: f70693a176b0410cac470f692003061a (127.9.155.66:34855)
I20260620 19:12:36.945978 24125 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.155.66@KRBTEST.COM'} at 127.9.155.66:52497
W20260620 19:12:37.038944 24453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:37.039215 24453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:37.039278 24453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:37.043313 24453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:37.043387 24453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:37.043471 24453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.67
I20260620 19:12:37.048799 24453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:36051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.67.keytab
--principal=kudu/127.9.155.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.67:44459
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.155.67
--webserver_port=45675
--tserver_master_addrs=127.9.155.126:33551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24453
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.67
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:12:37.049839 24453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:37.050844 24453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 19:12:37.060690 24453 server_base.cc:1034] running on GCE node
W20260620 19:12:37.060633 24459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:37.060925 24462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:37.060708 24460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:37.061409 24453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:37.062041 24453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:37.063213 24453 hybrid_clock.cc:648] HybridClock initialized: now 1781982757063196 us; error 31 us; skew 500 ppm
Jun 20 19:12:37 dist-test-slave-n3fx krb5kdc[23544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982757, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:37.065909 24453 init.cc:377] Logged in from keytab as kudu/127.9.155.67@KRBTEST.COM (short username kudu)
I20260620 19:12:37.066951 24453 webserver.cc:460] Webserver started at http://127.9.155.67:45675/ using document root <none> and password file <none>
I20260620 19:12:37.067518 24453 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:37.067610 24453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:12:37.071537 24453 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 19:12:37.073666 24469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:37.074961 24453 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 19:12:37.075083 24453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-2/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-2/wal
uuid: "037d4c71bd604f338452e010c6fd92f2"
format_stamp: "Formatted at 2026-06-20 19:12:35 on dist-test-slave-n3fx"
server_key: "e3e04ff741c4ff8b913153f78817607b"
server_key_iv: "bfb2376f4926a7c31508a84fa425eb44"
server_key_version: "encryptionkey@0"
I20260620 19:12:37.075628 24453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:37.091663 24453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:37.094475 24453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:37.094650 24453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:37.095203 24453 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:12:37.095996 24453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:12:37.096040 24453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:37.096092 24453 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:12:37.096119 24453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:37.105023 24453 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.67:44459
I20260620 19:12:37.105048 24582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.67:44459 every 8 connection(s)
I20260620 19:12:37.106045 24453 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
Jun 20 19:12:37 dist-test-slave-n3fx krb5kdc[23544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982757, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.67@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:12:37.115684 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 24453
I20260620 19:12:37.119635 24315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:37.108009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.67:48315 (local address 127.9.155.126:33551)
0620 19:12:37.108151 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:37.108155 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:37.109032 (+ 877us) server_negotiation.cc:407] Connection header received
0620 19:12:37.110270 (+ 1238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:37.110273 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:37.110331 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:37.110404 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:37.111567 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:37.112096 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:37.113262 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:37.113486 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:37.113575 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:37.116123 (+ 2548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:37.116126 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:37.116128 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:37.116156 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:37.117685 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:37.118227 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:37.118230 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:37.118231 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:37.118304 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:37.118625 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:37.118629 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:37.118631 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:37.118840 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:37.118929 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:37.119410 (+ 481us) server_negotiation.cc:299] Negotiation successful
0620 19:12:37.119513 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 19:12:37.120744 24585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:37.108433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:33551 (local address 127.9.155.67:48315)
0620 19:12:37.108881 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:37.108925 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:37.110065 (+ 1140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:37.110570 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:37.110577 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:37.110928 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:37.111427 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:37.111441 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:37.112241 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:37.112245 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:37.113135 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:37.113145 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:37.113632 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:37.113636 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:37.113805 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:37.114398 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:12:37.114414 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:37.115949 (+ 1535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:37.117788 (+ 1839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:37.117795 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:37.117812 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:37.118142 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:37.118380 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:37.118384 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:37.118386 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:37.118538 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:37.118931 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:37.118936 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:37.119170 (+ 234us) client_negotiation.cc:769] Sending connection context
0620 19:12:37.119356 (+ 186us) client_negotiation.cc:240] Negotiation successful
0620 19:12:37.119686 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":122,"threads_started":1}
I20260620 19:12:37.122076 24583 heartbeater.cc:347] Connected to a master server at 127.9.155.126:33551
I20260620 19:12:37.122356 24583 heartbeater.cc:464] Registering TS with master...
I20260620 19:12:37.122927 24583 heartbeater.cc:511] Master 127.9.155.126:33551 requested a full tablet report, sending...
I20260620 19:12:37.124049 24125 ts_manager.cc:194] Registered new tserver with Master: 037d4c71bd604f338452e010c6fd92f2 (127.9.155.67:44459)
I20260620 19:12:37.124468 24587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:37.116352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51840 (local address 127.9.155.126:33551)
0620 19:12:37.116613 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:37.116617 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:37.116653 (+ 36us) server_negotiation.cc:407] Connection header received
0620 19:12:37.116788 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:37.116792 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:37.116831 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:37.116901 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:37.117606 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:37.118329 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:37.119356 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:37.119675 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:37.119779 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:37.121310 (+ 1531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:37.121313 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:37.121315 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:37.121336 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:37.122742 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:37.123130 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:37.123134 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:37.123136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:37.123255 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:37.123555 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:37.123558 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:37.123559 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:37.123783 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:37.123958 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:37.124169 (+ 211us) server_negotiation.cc:299] Negotiation successful
0620 19:12:37.124337 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":417,"threads_started":1}
I20260620 19:12:37.124809 24125 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.155.67@KRBTEST.COM'} at 127.9.155.67:48315
I20260620 19:12:37.126591 9837 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:12:37.130417 24587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:37.127129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51844 (local address 127.9.155.126:33551)
0620 19:12:37.127236 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:37.127240 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:37.127429 (+ 189us) server_negotiation.cc:407] Connection header received
0620 19:12:37.127536 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:37.127586 (+ 50us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:37.127675 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:37.127753 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:12:37.127922 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:37.128464 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:37.129244 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:37.129463 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:37.129595 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:37.129747 (+ 152us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:12:37.130120 (+ 373us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:12:37.130203 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:37.130251 (+ 48us) server_negotiation.cc:299] Negotiation successful
0620 19:12:37.130300 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 19:12:37.130980 24125 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:51844:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:12:37.134135 24125 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:12:37.141661 24247 tablet_service.cc:1467] Processing CreateTablet for tablet 8ea5756386f643ffb365c87e6b5e8944 (DEFAULT_TABLE table=test-table [id=79a92a4db2344cc286527b087ee036d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:12:37.142311 24247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ea5756386f643ffb365c87e6b5e8944. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:12:37.144866 24205 tablet_bootstrap.cc:492] T 8ea5756386f643ffb365c87e6b5e8944 P 2b13fa8f898b4c99b52c87c89708aa37: Bootstrap starting.
I20260620 19:12:37.145584 24205 tablet_bootstrap.cc:654] T 8ea5756386f643ffb365c87e6b5e8944 P 2b13fa8f898b4c99b52c87c89708aa37: Neither blocks nor log segments found. Creating new log.
I20260620 19:12:37.146466 24320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:37.140739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.66:34855 (local address 127.0.0.1:36434)
0620 19:12:37.140862 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:37.140877 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:37.140954 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:37.142013 (+ 1059us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:37.142016 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:37.142046 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:12:37.142198 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:37.142206 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:37.143755 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:37.143758 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:37.144853 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:37.144860 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:37.145952 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:37.145956 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:37.146088 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:37.146139 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 19:12:37.146240 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 19:12:37.146328 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":17,"spinlock_wait_cycles":7168}
I20260620 19:12:37.146543 24593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:37.140970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.67:44459 (local address 127.0.0.1:46038)
0620 19:12:37.141190 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:37.141202 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:37.141277 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:37.142015 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:37.142018 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:37.142049 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:12:37.142361 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:37.142368 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:37.143514 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:37.143518 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:37.144761 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:37.144771 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:37.146144 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:37.146147 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:37.146241 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:37.146264 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 19:12:37.146329 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 19:12:37.146393 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":44,"spinlock_wait_cycles":5248,"thread_start_us":87,"threads_started":1}
I20260620 19:12:37.146960 24592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:37.140908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36434 (local address 127.9.155.66:34855)
0620 19:12:37.141504 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:37.141515 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:37.141537 (+ 22us) server_negotiation.cc:407] Connection header received
0620 19:12:37.141614 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:37.141620 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:37.141854 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:37.142164 (+ 310us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:12:37.142314 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:37.143632 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:37.144977 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:37.145806 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:37.145918 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:37.146068 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:37.146270 (+ 202us) server_negotiation.cc:299] Negotiation successful
0620 19:12:37.146369 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":143,"threads_started":1}
I20260620 19:12:37.147032 24594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:37.141338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46038 (local address 127.9.155.67:44459)
0620 19:12:37.141604 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:37.141614 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:37.141634 (+ 20us) server_negotiation.cc:407] Connection header received
0620 19:12:37.141696 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:37.141700 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:37.141851 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:37.142005 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:12:37.142481 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:37.143386 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:37.144903 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:37.145937 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:37.146030 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:37.146154 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:37.146795 (+ 641us) server_negotiation.cc:299] Negotiation successful
0620 19:12:37.146904 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":82,"threads_started":1}
I20260620 19:12:37.147310 24205 tablet_bootstrap.cc:492] T 8ea5756386f643ffb365c87e6b5e8944 P 2b13fa8f898b4c99b52c87c89708aa37: No bootstrap required, opened a new log
I20260620 19:12:37.147387 24205 ts_tablet_manager.cc:1397] T 8ea5756386f643ffb365c87e6b5e8944 P 2b13fa8f898b4c99b52c87c89708aa37: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 19:12:37.147776 24205 raft_consensus.cc:348] T 8ea5756386f643ffb365c87e6b5e8944 P 2b13fa8f898b4c99b52c87c89708aa37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f70693a176b0410cac470f692003061a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 34855 } } peers { permanent_uuid: "037d4c71bd604f338452e010c6fd92f2" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 44459 } } peers { permanent_uuid: "2b13fa8f898b4c99b52c87c89708aa37" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 46309 } }
I20260620 19:12:37.147902 24205 raft_consensus.cc:374] T 8ea5756386f643ffb365c87e6b5e8944 P 2b13fa8f898b4c99b52c87c89708aa37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:12:37.147936 24205 raft_consensus.cc:729] T 8ea5756386f643ffb365c87e6b5e8944 P 2b13fa8f898b4c99b52c87c89708aa37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b13fa8f898b4c99b52c87c89708aa37, State: Initialized, Role: FOLLOWER
I20260620 19:12:37.148052 24205 consensus_queue.cc:260] T 8ea5756386f643ffb365c87e6b5e8944 P 2b13fa8f898b4c99b52c87c89708aa37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f70693a176b0410cac470f692003061a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 34855 } } peers { permanent_uuid: "037d4c71bd604f338452e010c6fd92f2" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 44459 } } peers { permanent_uuid: "2b13fa8f898b4c99b52c87c89708aa37" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 46309 } }
I20260620 19:12:37.148335 24205 ts_tablet_manager.cc:1428] T 8ea5756386f643ffb365c87e6b5e8944 P 2b13fa8f898b4c99b52c87c89708aa37: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:12:37.149852 24384 tablet_service.cc:1467] Processing CreateTablet for tablet 8ea5756386f643ffb365c87e6b5e8944 (DEFAULT_TABLE table=test-table [id=79a92a4db2344cc286527b087ee036d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:12:37.149850 24517 tablet_service.cc:1467] Processing CreateTablet for tablet 8ea5756386f643ffb365c87e6b5e8944 (DEFAULT_TABLE table=test-table [id=79a92a4db2344cc286527b087ee036d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:12:37.150849 24384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ea5756386f643ffb365c87e6b5e8944. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:12:37.150858 24517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ea5756386f643ffb365c87e6b5e8944. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:12:37.157302 24596 tablet_bootstrap.cc:492] T 8ea5756386f643ffb365c87e6b5e8944 P 037d4c71bd604f338452e010c6fd92f2: Bootstrap starting.
I20260620 19:12:37.158214 24597 tablet_bootstrap.cc:492] T 8ea5756386f643ffb365c87e6b5e8944 P f70693a176b0410cac470f692003061a: Bootstrap starting.
I20260620 19:12:37.159967 24596 tablet_bootstrap.cc:654] T 8ea5756386f643ffb365c87e6b5e8944 P 037d4c71bd604f338452e010c6fd92f2: Neither blocks nor log segments found. Creating new log.
I20260620 19:12:37.160424 24597 tablet_bootstrap.cc:654] T 8ea5756386f643ffb365c87e6b5e8944 P f70693a176b0410cac470f692003061a: Neither blocks nor log segments found. Creating new log.
I20260620 19:12:37.160766 24596 log.cc:824] T 8ea5756386f643ffb365c87e6b5e8944 P 037d4c71bd604f338452e010c6fd92f2: Log is configured to *not* fsync() on all Append() calls
I20260620 19:12:37.161082 24597 log.cc:824] T 8ea5756386f643ffb365c87e6b5e8944 P f70693a176b0410cac470f692003061a: Log is configured to *not* fsync() on all Append() calls
I20260620 19:12:37.162667 24597 tablet_bootstrap.cc:492] T 8ea5756386f643ffb365c87e6b5e8944 P f70693a176b0410cac470f692003061a: No bootstrap required, opened a new log
I20260620 19:12:37.162674 24596 tablet_bootstrap.cc:492] T 8ea5756386f643ffb365c87e6b5e8944 P 037d4c71bd604f338452e010c6fd92f2: No bootstrap required, opened a new log
I20260620 19:12:37.162844 24596 ts_tablet_manager.cc:1397] T 8ea5756386f643ffb365c87e6b5e8944 P 037d4c71bd604f338452e010c6fd92f2: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.002s
I20260620 19:12:37.162844 24597 ts_tablet_manager.cc:1397] T 8ea5756386f643ffb365c87e6b5e8944 P f70693a176b0410cac470f692003061a: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 19:12:37.166031 24597 raft_consensus.cc:348] T 8ea5756386f643ffb365c87e6b5e8944 P f70693a176b0410cac470f692003061a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f70693a176b0410cac470f692003061a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 34855 } } peers { permanent_uuid: "037d4c71bd604f338452e010c6fd92f2" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 44459 } } peers { permanent_uuid: "2b13fa8f898b4c99b52c87c89708aa37" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 46309 } }
I20260620 19:12:37.166074 24596 raft_consensus.cc:348] T 8ea5756386f643ffb365c87e6b5e8944 P 037d4c71bd604f338452e010c6fd92f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f70693a176b0410cac470f692003061a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 34855 } } peers { permanent_uuid: "037d4c71bd604f338452e010c6fd92f2" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 44459 } } peers { permanent_uuid: "2b13fa8f898b4c99b52c87c89708aa37" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 46309 } }
I20260620 19:12:37.166280 24597 raft_consensus.cc:374] T 8ea5756386f643ffb365c87e6b5e8944 P f70693a176b0410cac470f692003061a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:12:37.166293 24596 raft_consensus.cc:374] T 8ea5756386f643ffb365c87e6b5e8944 P 037d4c71bd604f338452e010c6fd92f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:12:37.166316 24597 raft_consensus.cc:729] T 8ea5756386f643ffb365c87e6b5e8944 P f70693a176b0410cac470f692003061a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f70693a176b0410cac470f692003061a, State: Initialized, Role: FOLLOWER
I20260620 19:12:37.166357 24596 raft_consensus.cc:729] T 8ea5756386f643ffb365c87e6b5e8944 P 037d4c71bd604f338452e010c6fd92f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 037d4c71bd604f338452e010c6fd92f2, State: Initialized, Role: FOLLOWER
I20260620 19:12:37.166780 24597 consensus_queue.cc:260] T 8ea5756386f643ffb365c87e6b5e8944 P f70693a176b0410cac470f692003061a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f70693a176b0410cac470f692003061a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 34855 } } peers { permanent_uuid: "037d4c71bd604f338452e010c6fd92f2" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 44459 } } peers { permanent_uuid: "2b13fa8f898b4c99b52c87c89708aa37" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 46309 } }
I20260620 19:12:37.166798 24596 consensus_queue.cc:260] T 8ea5756386f643ffb365c87e6b5e8944 P 037d4c71bd604f338452e010c6fd92f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f70693a176b0410cac470f692003061a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 34855 } } peers { permanent_uuid: "037d4c71bd604f338452e010c6fd92f2" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 44459 } } peers { permanent_uuid: "2b13fa8f898b4c99b52c87c89708aa37" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 46309 } }
I20260620 19:12:37.167413 24451 heartbeater.cc:503] Master 127.9.155.126:33551 was elected leader, sending a full tablet report...
I20260620 19:12:37.167413 24583 heartbeater.cc:503] Master 127.9.155.126:33551 was elected leader, sending a full tablet report...
I20260620 19:12:37.167790 24597 ts_tablet_manager.cc:1428] T 8ea5756386f643ffb365c87e6b5e8944 P f70693a176b0410cac470f692003061a: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 19:12:37.167791 24596 ts_tablet_manager.cc:1428] T 8ea5756386f643ffb365c87e6b5e8944 P 037d4c71bd604f338452e010c6fd92f2: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
W20260620 19:12:37.178846 24452 tablet.cc:2369] T 8ea5756386f643ffb365c87e6b5e8944 P f70693a176b0410cac470f692003061a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:12:37.224828 24319 raft_consensus.cc:482] T 8ea5756386f643ffb365c87e6b5e8944 P 2b13fa8f898b4c99b52c87c89708aa37 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:12:37.225054 24319 raft_consensus.cc:504] T 8ea5756386f643ffb365c87e6b5e8944 P 2b13fa8f898b4c99b52c87c89708aa37 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f70693a176b0410cac470f692003061a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 34855 } } peers { permanent_uuid: "037d4c71bd604f338452e010c6fd92f2" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 44459 } } peers { permanent_uuid: "2b13fa8f898b4c99b52c87c89708aa37" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 46309 } }
I20260620 19:12:37.226131 24319 leader_election.cc:288] T 8ea5756386f643ffb365c87e6b5e8944 P 2b13fa8f898b4c99b52c87c89708aa37 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f70693a176b0410cac470f692003061a (127.9.155.66:34855), 037d4c71bd604f338452e010c6fd92f2 (127.9.155.67:44459)
W20260620 19:12:37.228122 24314 tablet.cc:2369] T 8ea5756386f643ffb365c87e6b5e8944 P 2b13fa8f898b4c99b52c87c89708aa37: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:12:37.230286 24592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:37.226499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.65:38645 (local address 127.9.155.66:34855)
0620 19:12:37.226643 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:37.226647 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:37.226947 (+ 300us) server_negotiation.cc:407] Connection header received
0620 19:12:37.227007 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:37.227010 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:37.227061 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:37.227143 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:12:37.227360 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:37.228142 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:37.229355 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:37.229758 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:37.229867 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:37.229915 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:37.230110 (+ 195us) server_negotiation.cc:299] Negotiation successful
0620 19:12:37.230177 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 19:12:37.230286 24594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:37.226499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.65:54733 (local address 127.9.155.67:44459)
0620 19:12:37.226641 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:37.226646 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:37.226731 (+ 85us) server_negotiation.cc:407] Connection header received
0620 19:12:37.226911 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:37.226915 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:37.226968 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:37.227073 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:12:37.227325 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:37.228119 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:37.229350 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:37.229749 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:37.229853 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:37.229907 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:37.230102 (+ 195us) server_negotiation.cc:299] Negotiation successful
0620 19:12:37.230168 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 19:12:37.230286 24316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:37.226407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.67:44459 (local address 127.9.155.65:54733)
0620 19:12:37.226643 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:37.226659 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:37.226780 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:37.227069 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:37.227071 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:37.227085 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:12:37.227210 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:37.227226 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:37.228264 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:37.228267 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:37.229214 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:37.229232 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:37.229867 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:37.229870 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:37.229968 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:37.230001 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 19:12:37.230063 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 19:12:37.230142 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":44}
I20260620 19:12:37.230305 24602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:37.226407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.66:34855 (local address 127.9.155.65:38645)
0620 19:12:37.226844 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:37.226857 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:37.226946 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:37.227153 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:37.227155 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:37.227167 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:12:37.227267 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:37.227273 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:37.228287 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:37.228291 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:37.229216 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:37.229225 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:37.229893 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:37.229896 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:37.229989 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:37.230006 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 19:12:37.230077 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 19:12:37.230145 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"mutex_wait_us":58,"thread_start_us":175,"threads_started":1}
I20260620 19:12:37.230899 24405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ea5756386f643ffb365c87e6b5e8944" candidate_uuid: "2b13fa8f898b4c99b52c87c89708aa37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f70693a176b0410cac470f692003061a" is_pre_election: true
I20260620 19:12:37.230916 24537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ea5756386f643ffb365c87e6b5e8944" candidate_uuid: "2b13fa8f898b4c99b52c87c89708aa37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "037d4c71bd604f338452e010c6fd92f2" is_pre_election: true
I20260620 19:12:37.231180 24405 raft_consensus.cc:2454] T 8ea5756386f643ffb365c87e6b5e8944 P f70693a176b0410cac470f692003061a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b13fa8f898b4c99b52c87c89708aa37 in term 0.
I20260620 19:12:37.231191 24537 raft_consensus.cc:2454] T 8ea5756386f643ffb365c87e6b5e8944 P 037d4c71bd604f338452e010c6fd92f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b13fa8f898b4c99b52c87c89708aa37 in term 0.
I20260620 19:12:37.231619 24201 leader_election.cc:302] T 8ea5756386f643ffb365c87e6b5e8944 P 2b13fa8f898b4c99b52c87c89708aa37 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b13fa8f898b4c99b52c87c89708aa37, f70693a176b0410cac470f692003061a; no voters:
I20260620 19:12:37.231876 24319 raft_consensus.cc:2781] T 8ea5756386f643ffb365c87e6b5e8944 P 2b13fa8f898b4c99b52c87c89708aa37 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:12:37.231948 24319 raft_consensus.cc:482] T 8ea5756386f643ffb365c87e6b5e8944 P 2b13fa8f898b4c99b52c87c89708aa37 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:12:37.231977 24319 raft_consensus.cc:3037] T 8ea5756386f643ffb365c87e6b5e8944 P 2b13fa8f898b4c99b52c87c89708aa37 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:12:37.232980 24319 raft_consensus.cc:504] T 8ea5756386f643ffb365c87e6b5e8944 P 2b13fa8f898b4c99b52c87c89708aa37 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f70693a176b0410cac470f692003061a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 34855 } } peers { permanent_uuid: "037d4c71bd604f338452e010c6fd92f2" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 44459 } } peers { permanent_uuid: "2b13fa8f898b4c99b52c87c89708aa37" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 46309 } }
I20260620 19:12:37.233310 24319 leader_election.cc:288] T 8ea5756386f643ffb365c87e6b5e8944 P 2b13fa8f898b4c99b52c87c89708aa37 [CANDIDATE]: Term 1 election: Requested vote from peers f70693a176b0410cac470f692003061a (127.9.155.66:34855), 037d4c71bd604f338452e010c6fd92f2 (127.9.155.67:44459)
I20260620 19:12:37.233654 24537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ea5756386f643ffb365c87e6b5e8944" candidate_uuid: "2b13fa8f898b4c99b52c87c89708aa37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "037d4c71bd604f338452e010c6fd92f2"
I20260620 19:12:37.233777 24537 raft_consensus.cc:3037] T 8ea5756386f643ffb365c87e6b5e8944 P 037d4c71bd604f338452e010c6fd92f2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:12:37.233785 24405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ea5756386f643ffb365c87e6b5e8944" candidate_uuid: "2b13fa8f898b4c99b52c87c89708aa37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f70693a176b0410cac470f692003061a"
I20260620 19:12:37.233907 24405 raft_consensus.cc:3037] T 8ea5756386f643ffb365c87e6b5e8944 P f70693a176b0410cac470f692003061a [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:12:37.234802 24537 raft_consensus.cc:2454] T 8ea5756386f643ffb365c87e6b5e8944 P 037d4c71bd604f338452e010c6fd92f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b13fa8f898b4c99b52c87c89708aa37 in term 1.
I20260620 19:12:37.234843 24405 raft_consensus.cc:2454] T 8ea5756386f643ffb365c87e6b5e8944 P f70693a176b0410cac470f692003061a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b13fa8f898b4c99b52c87c89708aa37 in term 1.
I20260620 19:12:37.235081 24202 leader_election.cc:302] T 8ea5756386f643ffb365c87e6b5e8944 P 2b13fa8f898b4c99b52c87c89708aa37 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 037d4c71bd604f338452e010c6fd92f2, 2b13fa8f898b4c99b52c87c89708aa37; no voters:
I20260620 19:12:37.235258 24319 raft_consensus.cc:2781] T 8ea5756386f643ffb365c87e6b5e8944 P 2b13fa8f898b4c99b52c87c89708aa37 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:12:37.235463 24319 raft_consensus.cc:686] T 8ea5756386f643ffb365c87e6b5e8944 P 2b13fa8f898b4c99b52c87c89708aa37 [term 1 LEADER]: Becoming Leader. State: Replica: 2b13fa8f898b4c99b52c87c89708aa37, State: Running, Role: LEADER
I20260620 19:12:37.235692 24319 consensus_queue.cc:237] T 8ea5756386f643ffb365c87e6b5e8944 P 2b13fa8f898b4c99b52c87c89708aa37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f70693a176b0410cac470f692003061a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 34855 } } peers { permanent_uuid: "037d4c71bd604f338452e010c6fd92f2" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 44459 } } peers { permanent_uuid: "2b13fa8f898b4c99b52c87c89708aa37" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 46309 } }
I20260620 19:12:37.238507 24125 catalog_manager.cc:5495] T 8ea5756386f643ffb365c87e6b5e8944 P 2b13fa8f898b4c99b52c87c89708aa37 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2b13fa8f898b4c99b52c87c89708aa37 (127.9.155.65). New cstate: current_term: 1 leader_uuid: "2b13fa8f898b4c99b52c87c89708aa37" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f70693a176b0410cac470f692003061a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 34855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "037d4c71bd604f338452e010c6fd92f2" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 44459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b13fa8f898b4c99b52c87c89708aa37" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 46309 } health_report { overall_health: HEALTHY } } }
I20260620 19:12:37.272127 24605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:37.268188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33208 (local address 127.9.155.65:46309)
0620 19:12:37.268489 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:37.268493 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:37.268505 (+ 12us) server_negotiation.cc:407] Connection header received
0620 19:12:37.268586 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:37.268589 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:37.268650 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:37.268774 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:12:37.268993 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:37.269633 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:37.270488 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:37.270823 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:37.270922 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:37.271224 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:12:37.271731 (+ 507us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:12:37.271826 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:37.271913 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 19:12:37.271970 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":103,"threads_started":1}
I20260620 19:12:37.276016 24405 raft_consensus.cc:1261] T 8ea5756386f643ffb365c87e6b5e8944 P f70693a176b0410cac470f692003061a [term 1 FOLLOWER]: Refusing update from remote peer 2b13fa8f898b4c99b52c87c89708aa37: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:12:37.276016 24537 raft_consensus.cc:1261] T 8ea5756386f643ffb365c87e6b5e8944 P 037d4c71bd604f338452e010c6fd92f2 [term 1 FOLLOWER]: Refusing update from remote peer 2b13fa8f898b4c99b52c87c89708aa37: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:12:37.276841 24603 consensus_queue.cc:1035] T 8ea5756386f643ffb365c87e6b5e8944 P 2b13fa8f898b4c99b52c87c89708aa37 [LEADER]: Connected to new peer: Peer: permanent_uuid: "037d4c71bd604f338452e010c6fd92f2" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 44459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:12:37.277027 24319 consensus_queue.cc:1035] T 8ea5756386f643ffb365c87e6b5e8944 P 2b13fa8f898b4c99b52c87c89708aa37 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f70693a176b0410cac470f692003061a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 34855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:12:37.285868 24606 mvcc.cc:205] Tried to move back new op lower bound from 7299001373796114432 to 7299001373639897088. Current Snapshot: MvccSnapshot[applied={T|T < 7299001373796114432}]
I20260620 19:12:37.293890 24125 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:51844:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:12:37.294099 24125 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:51844:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:12:37.296805 24125 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b34ec788c4684b4bb78bf6fed9657a8c: Sending DeleteTablet for 3 replicas of tablet 8ea5756386f643ffb365c87e6b5e8944
I20260620 19:12:37.297652 24517 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ea5756386f643ffb365c87e6b5e8944 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:12:37 UTC) from {username='kudu', principal='kudu/127.9.155.126@KRBTEST.COM'} at 127.0.0.1:46038
I20260620 19:12:37.297814 24247 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ea5756386f643ffb365c87e6b5e8944 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:12:37 UTC) from {username='kudu', principal='kudu/127.9.155.126@KRBTEST.COM'} at 127.0.0.1:33198
I20260620 19:12:37.297922 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 24163
I20260620 19:12:37.298235 24384 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ea5756386f643ffb365c87e6b5e8944 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:12:37 UTC) from {username='kudu', principal='kudu/127.9.155.126@KRBTEST.COM'} at 127.0.0.1:36434
I20260620 19:12:37.298604 24616 tablet_replica.cc:330] T 8ea5756386f643ffb365c87e6b5e8944 P 037d4c71bd604f338452e010c6fd92f2: stopping tablet replica
I20260620 19:12:37.298964 24616 raft_consensus.cc:2229] T 8ea5756386f643ffb365c87e6b5e8944 P 037d4c71bd604f338452e010c6fd92f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:12:37.299316 24616 raft_consensus.cc:2258] T 8ea5756386f643ffb365c87e6b5e8944 P 037d4c71bd604f338452e010c6fd92f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:12:37.299690 24617 tablet_replica.cc:330] T 8ea5756386f643ffb365c87e6b5e8944 P f70693a176b0410cac470f692003061a: stopping tablet replica
I20260620 19:12:37.300006 24617 raft_consensus.cc:2229] T 8ea5756386f643ffb365c87e6b5e8944 P f70693a176b0410cac470f692003061a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:12:37.300355 24617 raft_consensus.cc:2258] T 8ea5756386f643ffb365c87e6b5e8944 P f70693a176b0410cac470f692003061a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:12:37.301244 24616 ts_tablet_manager.cc:1905] T 8ea5756386f643ffb365c87e6b5e8944 P 037d4c71bd604f338452e010c6fd92f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:12:37.301599 24617 ts_tablet_manager.cc:1905] T 8ea5756386f643ffb365c87e6b5e8944 P f70693a176b0410cac470f692003061a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:12:37.303789 24617 ts_tablet_manager.cc:1918] T 8ea5756386f643ffb365c87e6b5e8944 P f70693a176b0410cac470f692003061a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:12:37.303902 24617 log.cc:1196] T 8ea5756386f643ffb365c87e6b5e8944 P f70693a176b0410cac470f692003061a: Deleting WAL directory at /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-1/wal/wals/8ea5756386f643ffb365c87e6b5e8944
I20260620 19:12:37.303984 24616 ts_tablet_manager.cc:1918] T 8ea5756386f643ffb365c87e6b5e8944 P 037d4c71bd604f338452e010c6fd92f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:12:37.304103 24616 log.cc:1196] T 8ea5756386f643ffb365c87e6b5e8944 P 037d4c71bd604f338452e010c6fd92f2: Deleting WAL directory at /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-2/wal/wals/8ea5756386f643ffb365c87e6b5e8944
I20260620 19:12:37.304291 24617 ts_tablet_manager.cc:1939] T 8ea5756386f643ffb365c87e6b5e8944 P f70693a176b0410cac470f692003061a: Deleting consensus metadata
I20260620 19:12:37.304431 24616 ts_tablet_manager.cc:1939] T 8ea5756386f643ffb365c87e6b5e8944 P 037d4c71bd604f338452e010c6fd92f2: Deleting consensus metadata
I20260620 19:12:37.305335 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 24318
W20260620 19:12:37.305311 24110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.155.65:46309 (error 108)
I20260620 19:12:37.306749 24593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:37.306456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.65:46309 (local address 127.0.0.1:33212)
0620 19:12:37.306598 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:37.306672 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.65:46309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 19:12:37.307155 24112 catalog_manager.cc:4841] TS 037d4c71bd604f338452e010c6fd92f2 (127.9.155.67:44459): tablet 8ea5756386f643ffb365c87e6b5e8944 (table test-table [id=79a92a4db2344cc286527b087ee036d1]) successfully deleted
W20260620 19:12:37.306910 24110 catalog_manager.cc:4584] TS 2b13fa8f898b4c99b52c87c89708aa37 (127.9.155.65:46309): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8ea5756386f643ffb365c87e6b5e8944: Network error: Client connection negotiation failed: client connection to 127.9.155.65:46309: connect: Connection refused (error 111)
I20260620 19:12:37.310810 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 24453
I20260620 19:12:37.311594 24593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:37.311092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.66:34855 (local address 127.0.0.1:36438)
0620 19:12:37.311442 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:37.311500 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.66:34855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260}
I20260620 19:12:37.315797 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 24092
I20260620 19:12:37.321540 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:33551
--webserver_interface=127.9.155.126
--webserver_port=40697
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab
--principal=kudu/127.9.155.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:36051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.155.126:33551
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:12:37.431147 24619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:37.431458 24619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:37.431500 24619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:37.435650 24619 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:12:37.435734 24619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:37.435752 24619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:37.435767 24619 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:12:37.435782 24619 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:12:37.441365 24619 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:36051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.155.126:33551
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab
--principal=kudu/127.9.155.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:33551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.155.126
--webserver_port=40697
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24619
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:12:37.442463 24619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:37.443451 24619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:12:37.452304 24627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:37.452335 24625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:37.452335 24624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:37.452862 24619 server_base.cc:1034] running on GCE node
I20260620 19:12:37.453444 24619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:37.454605 24619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:37.455804 24619 hybrid_clock.cc:648] HybridClock initialized: now 1781982757455784 us; error 38 us; skew 500 ppm
Jun 20 19:12:37 dist-test-slave-n3fx krb5kdc[23544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982757, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:37.458748 24619 init.cc:377] Logged in from keytab as kudu/127.9.155.126@KRBTEST.COM (short username kudu)
I20260620 19:12:37.459653 24619 webserver.cc:460] Webserver started at http://127.9.155.126:40697/ using document root <none> and password file <none>
I20260620 19:12:37.460178 24619 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:37.460270 24619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:12:37.464126 24619 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260620 19:12:37.466288 24634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:37.467607 24619 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 19:12:37.467716 24619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/master-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/master-0/wal
uuid: "b34ec788c4684b4bb78bf6fed9657a8c"
format_stamp: "Formatted at 2026-06-20 19:12:35 on dist-test-slave-n3fx"
server_key: "4e368b3c54979a8b4297115bc51f9dcd"
server_key_iv: "cbad7a6c1833629c473316f986ab7d22"
server_key_version: "encryptionkey@0"
I20260620 19:12:37.468367 24619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:37.493046 24619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:37.496119 24619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:37.496330 24619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:37.503552 24619 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.126:33551
I20260620 19:12:37.503782 24686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.126:33551 every 8 connection(s)
I20260620 19:12:37.504675 24619 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
I20260620 19:12:37.507797 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 24619
I20260620 19:12:37.508181 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.65:46309
--local_ip_for_outbound_sockets=127.9.155.65
--tserver_master_addrs=127.9.155.126:33551
--webserver_port=40617
--webserver_interface=127.9.155.65
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.65.keytab
--principal=kudu/127.9.155.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:36051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:12:37.509227 24687 sys_catalog.cc:263] Verifying existing consensus state
I20260620 19:12:37.511004 24687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b34ec788c4684b4bb78bf6fed9657a8c: Bootstrap starting.
I20260620 19:12:37.528123 24687 log.cc:824] T 00000000000000000000000000000000 P b34ec788c4684b4bb78bf6fed9657a8c: Log is configured to *not* fsync() on all Append() calls
I20260620 19:12:37.539870 24687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b34ec788c4684b4bb78bf6fed9657a8c: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:12:37.540329 24687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b34ec788c4684b4bb78bf6fed9657a8c: Bootstrap complete.
I20260620 19:12:37.544183 24687 raft_consensus.cc:348] T 00000000000000000000000000000000 P b34ec788c4684b4bb78bf6fed9657a8c [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b34ec788c4684b4bb78bf6fed9657a8c" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 33551 } }
I20260620 19:12:37.544651 24687 raft_consensus.cc:729] T 00000000000000000000000000000000 P b34ec788c4684b4bb78bf6fed9657a8c [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: b34ec788c4684b4bb78bf6fed9657a8c, State: Initialized, Role: FOLLOWER
I20260620 19:12:37.545120 24687 consensus_queue.cc:260] T 00000000000000000000000000000000 P b34ec788c4684b4bb78bf6fed9657a8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b34ec788c4684b4bb78bf6fed9657a8c" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 33551 } }
I20260620 19:12:37.545241 24687 raft_consensus.cc:388] T 00000000000000000000000000000000 P b34ec788c4684b4bb78bf6fed9657a8c [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:12:37.545308 24687 raft_consensus.cc:482] T 00000000000000000000000000000000 P b34ec788c4684b4bb78bf6fed9657a8c [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:12:37.545399 24687 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b34ec788c4684b4bb78bf6fed9657a8c [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:12:37.547446 24687 raft_consensus.cc:504] T 00000000000000000000000000000000 P b34ec788c4684b4bb78bf6fed9657a8c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b34ec788c4684b4bb78bf6fed9657a8c" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 33551 } }
I20260620 19:12:37.547771 24687 leader_election.cc:302] T 00000000000000000000000000000000 P b34ec788c4684b4bb78bf6fed9657a8c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b34ec788c4684b4bb78bf6fed9657a8c; no voters:
I20260620 19:12:37.548080 24687 leader_election.cc:288] T 00000000000000000000000000000000 P b34ec788c4684b4bb78bf6fed9657a8c [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 19:12:37.548194 24691 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b34ec788c4684b4bb78bf6fed9657a8c [term 3 FOLLOWER]: Leader election won for term 3
I20260620 19:12:37.548413 24691 raft_consensus.cc:686] T 00000000000000000000000000000000 P b34ec788c4684b4bb78bf6fed9657a8c [term 3 LEADER]: Becoming Leader. State: Replica: b34ec788c4684b4bb78bf6fed9657a8c, State: Running, Role: LEADER
I20260620 19:12:37.548718 24691 consensus_queue.cc:237] T 00000000000000000000000000000000 P b34ec788c4684b4bb78bf6fed9657a8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b34ec788c4684b4bb78bf6fed9657a8c" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 33551 } }
I20260620 19:12:37.549364 24687 sys_catalog.cc:564] T 00000000000000000000000000000000 P b34ec788c4684b4bb78bf6fed9657a8c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:12:37.550091 24692 sys_catalog.cc:455] T 00000000000000000000000000000000 P b34ec788c4684b4bb78bf6fed9657a8c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "b34ec788c4684b4bb78bf6fed9657a8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b34ec788c4684b4bb78bf6fed9657a8c" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 33551 } } }
I20260620 19:12:37.550231 24692 sys_catalog.cc:458] T 00000000000000000000000000000000 P b34ec788c4684b4bb78bf6fed9657a8c [sys.catalog]: This master's current role is: LEADER
I20260620 19:12:37.550837 24693 sys_catalog.cc:455] T 00000000000000000000000000000000 P b34ec788c4684b4bb78bf6fed9657a8c [sys.catalog]: SysCatalogTable state changed. Reason: New leader b34ec788c4684b4bb78bf6fed9657a8c. Latest consensus state: current_term: 3 leader_uuid: "b34ec788c4684b4bb78bf6fed9657a8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b34ec788c4684b4bb78bf6fed9657a8c" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 33551 } } }
I20260620 19:12:37.550979 24693 sys_catalog.cc:458] T 00000000000000000000000000000000 P b34ec788c4684b4bb78bf6fed9657a8c [sys.catalog]: This master's current role is: LEADER
I20260620 19:12:37.551189 24700 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:12:37.556546 24700 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:12:37.557217 24700 catalog_manager.cc:1259] Loaded cluster ID: d9a35173f7a344e9b05224cb361a8f31
I20260620 19:12:37.557284 24700 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260620 19:12:37.558372 24700 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f984f49c9ad google::LogMessage::Fail() at ??:0
@ 0x7f984f49f88a google::LogMessage::SendToLog() at ??:0
@ 0x7f984f49c4de google::LogMessage::Flush() at ??:0
@ 0x7f984f49da19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f9859ddbfdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f9859ddcd2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f9859dd9579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f9859e1d640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f98516ee3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f98516eec83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f98516f03f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f98516df512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f9852d236db start_thread at ??:0
@ 0x7f984df9a61f clone at ??:0
W20260620 19:12:37.634676 24689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:37.634919 24689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:37.635004 24689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:37.639117 24689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:37.639178 24689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:37.639241 24689 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.65
I20260620 19:12:37.644667 24689 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:36051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.65.keytab
--principal=kudu/127.9.155.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.65:46309
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.155.65
--webserver_port=40617
--tserver_master_addrs=127.9.155.126:33551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24689
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.65
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:12:37.645710 24689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:37.646696 24689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:12:37.655907 24728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:37.655907 24729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:37.656112 24689 server_base.cc:1034] running on GCE node
W20260620 19:12:37.656199 24731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:37.656666 24689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:37.657295 24689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:37.658439 24689 hybrid_clock.cc:648] HybridClock initialized: now 1781982757658398 us; error 58 us; skew 500 ppm
Jun 20 19:12:37 dist-test-slave-n3fx krb5kdc[23544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982757, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:37.661648 24689 init.cc:377] Logged in from keytab as kudu/127.9.155.65@KRBTEST.COM (short username kudu)
I20260620 19:12:37.662695 24689 webserver.cc:460] Webserver started at http://127.9.155.65:40617/ using document root <none> and password file <none>
I20260620 19:12:37.663352 24689 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:37.663434 24689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:12:37.668118 24689 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20260620 19:12:37.670858 24738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:37.672391 24689 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:12:37.672521 24689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-0/wal
uuid: "2b13fa8f898b4c99b52c87c89708aa37"
format_stamp: "Formatted at 2026-06-20 19:12:35 on dist-test-slave-n3fx"
server_key: "6a53c726ff4021dba8d7ac3f9460e48a"
server_key_iv: "ab9796f5b048326491fe0833c8761fd0"
server_key_version: "encryptionkey@0"
I20260620 19:12:37.673242 24689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:37.702592 24689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:37.706308 24689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:37.706506 24689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:37.707341 24689 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:12:37.708839 24745 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 19:12:37.713008 24689 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 19:12:37.713094 24689 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260620 19:12:37.713162 24689 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 19:12:37.715440 24689 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 19:12:37.715510 24689 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260620 19:12:37.715806 24745 tablet_bootstrap.cc:492] T 8ea5756386f643ffb365c87e6b5e8944 P 2b13fa8f898b4c99b52c87c89708aa37: Bootstrap starting.
I20260620 19:12:37.726130 24689 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.65:46309
I20260620 19:12:37.726230 24852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.65:46309 every 8 connection(s)
I20260620 19:12:37.727171 24689 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
I20260620 19:12:37.734947 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 24689
I20260620 19:12:37.735354 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.66:34855
--local_ip_for_outbound_sockets=127.9.155.66
--tserver_master_addrs=127.9.155.126:33551
--webserver_port=33871
--webserver_interface=127.9.155.66
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.66.keytab
--principal=kudu/127.9.155.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:36051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:12:37.736269 24745 log.cc:824] T 8ea5756386f643ffb365c87e6b5e8944 P 2b13fa8f898b4c99b52c87c89708aa37: Log is configured to *not* fsync() on all Append() calls
I20260620 19:12:37.745148 24745 tablet_bootstrap.cc:492] T 8ea5756386f643ffb365c87e6b5e8944 P 2b13fa8f898b4c99b52c87c89708aa37: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:12:37.745564 24745 tablet_bootstrap.cc:492] T 8ea5756386f643ffb365c87e6b5e8944 P 2b13fa8f898b4c99b52c87c89708aa37: Bootstrap complete.
I20260620 19:12:37.746933 24745 ts_tablet_manager.cc:1397] T 8ea5756386f643ffb365c87e6b5e8944 P 2b13fa8f898b4c99b52c87c89708aa37: Time spent bootstrapping tablet: real 0.031s user 0.021s sys 0.007s
I20260620 19:12:37.750265 24745 raft_consensus.cc:348] T 8ea5756386f643ffb365c87e6b5e8944 P 2b13fa8f898b4c99b52c87c89708aa37 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f70693a176b0410cac470f692003061a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 34855 } } peers { permanent_uuid: "037d4c71bd604f338452e010c6fd92f2" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 44459 } } peers { permanent_uuid: "2b13fa8f898b4c99b52c87c89708aa37" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 46309 } }
I20260620 19:12:37.750842 24745 raft_consensus.cc:729] T 8ea5756386f643ffb365c87e6b5e8944 P 2b13fa8f898b4c99b52c87c89708aa37 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b13fa8f898b4c99b52c87c89708aa37, State: Initialized, Role: FOLLOWER
I20260620 19:12:37.751374 24745 consensus_queue.cc:260] T 8ea5756386f643ffb365c87e6b5e8944 P 2b13fa8f898b4c99b52c87c89708aa37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f70693a176b0410cac470f692003061a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 34855 } } peers { permanent_uuid: "037d4c71bd604f338452e010c6fd92f2" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 44459 } } peers { permanent_uuid: "2b13fa8f898b4c99b52c87c89708aa37" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 46309 } }
I20260620 19:12:37.752496 24745 ts_tablet_manager.cc:1428] T 8ea5756386f643ffb365c87e6b5e8944 P 2b13fa8f898b4c99b52c87c89708aa37: Time spent starting tablet: real 0.005s user 0.007s sys 0.001s
W20260620 19:12:37.797384 24855 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:12:37.729147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:33551 (local address 127.9.155.65:44119)
0620 19:12:37.729619 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:37.729672 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:37.730512 (+ 840us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:37.796592 (+ 66080us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:33551: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":122,"threads_started":1}
W20260620 19:12:37.797833 24853 heartbeater.cc:644] Failed to heartbeat to 127.9.155.126:33551 (0 consecutive failures): Network error: Failed to ping master at 127.9.155.126:33551: Client connection negotiation failed: client connection to 127.9.155.126:33551: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 19:12:37.798544 24855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:37.798309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:33551 (local address 127.9.155.65:56143)
0620 19:12:37.798428 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:37.798490 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 19:12:37.799175 24855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:37.798974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:33551 (local address 127.9.155.65:47685)
0620 19:12:37.799091 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:37.799131 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 19:12:37.799336 24853 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 19:12:37.852830 24856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:37.853076 24856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:37.853137 24856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:37.857223 24856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:37.857295 24856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:37.857373 24856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.66
I20260620 19:12:37.862870 24856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:36051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.66.keytab
--principal=kudu/127.9.155.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.66:34855
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.155.66
--webserver_port=33871
--tserver_master_addrs=127.9.155.126:33551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.66
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:12:37.863950 24856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:37.864990 24856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:12:37.874943 24866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:37.874944 24864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:37.874989 24863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:37.875234 24856 server_base.cc:1034] running on GCE node
I20260620 19:12:37.875734 24856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:37.876454 24856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:37.877642 24856 hybrid_clock.cc:648] HybridClock initialized: now 1781982757877613 us; error 46 us; skew 500 ppm
Jun 20 19:12:37 dist-test-slave-n3fx krb5kdc[23544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982757, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:37.880543 24856 init.cc:377] Logged in from keytab as kudu/127.9.155.66@KRBTEST.COM (short username kudu)
I20260620 19:12:37.881507 24856 webserver.cc:460] Webserver started at http://127.9.155.66:33871/ using document root <none> and password file <none>
I20260620 19:12:37.882052 24856 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:37.882114 24856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:12:37.886030 24856 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 19:12:37.888195 24873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:37.889501 24856 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 19:12:37.889612 24856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-1/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-1/wal
uuid: "f70693a176b0410cac470f692003061a"
format_stamp: "Formatted at 2026-06-20 19:12:35 on dist-test-slave-n3fx"
server_key: "5f518b7d8c2a8e6aae061b4da68b21af"
server_key_iv: "ef32bf5b29d8a758a29972e0a86100dc"
server_key_version: "encryptionkey@0"
I20260620 19:12:37.890156 24856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:37.905900 24856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:37.909158 24856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:37.909340 24856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:37.909935 24856 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:12:37.910777 24856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:12:37.910825 24856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:37.910876 24856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:12:37.910904 24856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:37.919450 24856 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.66:34855
I20260620 19:12:37.919471 24986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.66:34855 every 8 connection(s)
I20260620 19:12:37.920528 24856 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
I20260620 19:12:37.924328 24989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:37.922603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:33551 (local address 127.9.155.66:36153)
0620 19:12:37.923050 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:37.923157 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":116,"threads_started":1}
I20260620 19:12:37.924620 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 24856
W20260620 19:12:37.924777 24987 heartbeater.cc:644] Failed to heartbeat to 127.9.155.126:33551 (0 consecutive failures): Network error: Failed to ping master at 127.9.155.126:33551: Client connection negotiation failed: client connection to 127.9.155.126:33551: connect: Connection refused (error 111)
I20260620 19:12:37.925030 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.67:44459
--local_ip_for_outbound_sockets=127.9.155.67
--tserver_master_addrs=127.9.155.126:33551
--webserver_port=45675
--webserver_interface=127.9.155.67
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.67.keytab
--principal=kudu/127.9.155.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:36051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:12:37.925395 24989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:37.925147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:33551 (local address 127.9.155.66:39389)
0620 19:12:37.925300 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:37.925344 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260620 19:12:37.925937 24989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:37.925749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:33551 (local address 127.9.155.66:35039)
0620 19:12:37.925844 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:37.925889 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 19:12:37.926110 24987 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 19:12:38.035728 24990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:38.036007 24990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:38.036052 24990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:38.040135 24990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:38.040205 24990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:38.040283 24990 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.67
I20260620 19:12:38.045574 24990 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:36051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.67.keytab
--principal=kudu/127.9.155.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.67:44459
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.155.67
--webserver_port=45675
--tserver_master_addrs=127.9.155.126:33551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24990
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.67
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:12:38.046669 24990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:38.047685 24990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:12:38.057060 24998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:38.057201 24990 server_base.cc:1034] running on GCE node
W20260620 19:12:38.057054 24995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:38.057372 24996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:38.057761 24990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:38.058413 24990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:38.059584 24990 hybrid_clock.cc:648] HybridClock initialized: now 1781982758059564 us; error 32 us; skew 500 ppm
Jun 20 19:12:38 dist-test-slave-n3fx krb5kdc[23544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982758, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:38.062515 24990 init.cc:377] Logged in from keytab as kudu/127.9.155.67@KRBTEST.COM (short username kudu)
I20260620 19:12:38.063468 24990 webserver.cc:460] Webserver started at http://127.9.155.67:45675/ using document root <none> and password file <none>
I20260620 19:12:38.064020 24990 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:38.064090 24990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:12:38.068190 24990 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.002s
I20260620 19:12:38.070499 25005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:38.071893 24990 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 19:12:38.072018 24990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-2/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-2/wal
uuid: "037d4c71bd604f338452e010c6fd92f2"
format_stamp: "Formatted at 2026-06-20 19:12:35 on dist-test-slave-n3fx"
server_key: "e3e04ff741c4ff8b913153f78817607b"
server_key_iv: "bfb2376f4926a7c31508a84fa425eb44"
server_key_version: "encryptionkey@0"
I20260620 19:12:38.072633 24990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:38.087782 24990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:38.091070 24990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:38.091264 24990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:38.091871 24990 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:12:38.092770 24990 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:12:38.092828 24990 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:38.092882 24990 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:12:38.092907 24990 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:38.102030 24990 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.67:44459
I20260620 19:12:38.102057 25118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.67:44459 every 8 connection(s)
I20260620 19:12:38.103137 24990 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
I20260620 19:12:38.107290 25121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:38.105465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:33551 (local address 127.9.155.67:38633)
0620 19:12:38.105957 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:38.106072 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":125,"threads_started":1}
W20260620 19:12:38.107823 25119 heartbeater.cc:644] Failed to heartbeat to 127.9.155.126:33551 (0 consecutive failures): Network error: Failed to ping master at 127.9.155.126:33551: Client connection negotiation failed: client connection to 127.9.155.126:33551: connect: Connection refused (error 111)
I20260620 19:12:38.108525 25121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:38.108281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:33551 (local address 127.9.155.67:56745)
0620 19:12:38.108403 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:38.108459 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 19:12:38.109203 25121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:38.109012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:33551 (local address 127.9.155.67:38761)
0620 19:12:38.109106 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:38.109149 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 19:12:38.109395 25119 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 19:12:38.111241 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 24990
I20260620 19:12:38.800483 25123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:38.799980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:33551 (local address 127.9.155.65:50789)
0620 19:12:38.800294 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:38.800379 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":77,"threads_started":1}
I20260620 19:12:38.927122 25124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:38.926687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:33551 (local address 127.9.155.66:49993)
0620 19:12:38.926952 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:38.927013 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":70,"threads_started":1}
I20260620 19:12:39.067291 25125 raft_consensus.cc:482] T 8ea5756386f643ffb365c87e6b5e8944 P 2b13fa8f898b4c99b52c87c89708aa37 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:12:39.067492 25125 raft_consensus.cc:504] T 8ea5756386f643ffb365c87e6b5e8944 P 2b13fa8f898b4c99b52c87c89708aa37 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f70693a176b0410cac470f692003061a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 34855 } } peers { permanent_uuid: "037d4c71bd604f338452e010c6fd92f2" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 44459 } } peers { permanent_uuid: "2b13fa8f898b4c99b52c87c89708aa37" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 46309 } }
I20260620 19:12:39.068869 25125 leader_election.cc:288] T 8ea5756386f643ffb365c87e6b5e8944 P 2b13fa8f898b4c99b52c87c89708aa37 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f70693a176b0410cac470f692003061a (127.9.155.66:34855), 037d4c71bd604f338452e010c6fd92f2 (127.9.155.67:44459)
Jun 20 19:12:39 dist-test-slave-n3fx krb5kdc[23544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982757, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.65@KRBTEST.COM for kudu/127.9.155.67@KRBTEST.COM
Jun 20 19:12:39 dist-test-slave-n3fx krb5kdc[23544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982757, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.65@KRBTEST.COM for kudu/127.9.155.66@KRBTEST.COM
I20260620 19:12:39.082741 25126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:39.069172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.67:44459 (local address 127.9.155.65:47359)
0620 19:12:39.069685 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:39.069699 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:39.069839 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:39.072001 (+ 2162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:39.072003 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:39.072180 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:39.072671 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:39.072687 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:39.073950 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:39.073952 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:39.074935 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:39.074944 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:39.075620 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:39.075624 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:39.075843 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:39.076539 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:12:39.076553 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:39.078231 (+ 1678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:39.080367 (+ 2136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:39.080374 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:39.080388 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:39.080759 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:39.081079 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:39.081082 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:39.081084 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:39.081211 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:39.081812 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:39.081818 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:39.082042 (+ 224us) client_negotiation.cc:769] Sending connection context
0620 19:12:39.082217 (+ 175us) client_negotiation.cc:240] Negotiation successful
0620 19:12:39.082536 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"mutex_wait_us":338,"spinlock_wait_cycles":28800,"thread_start_us":79,"threads_started":1}
I20260620 19:12:39.082973 25127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:39.069354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.65:50065 (local address 127.9.155.66:34855)
0620 19:12:39.069763 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:39.069790 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:39.069834 (+ 44us) server_negotiation.cc:407] Connection header received
0620 19:12:39.070731 (+ 897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:39.070739 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:39.071201 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:39.071532 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:39.072838 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:39.073825 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:39.075047 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:39.075509 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:39.075659 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:39.078674 (+ 3015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:39.078678 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:39.078692 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:39.078726 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:39.080389 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:39.080852 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:39.080858 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:39.080864 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:39.080975 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:39.081288 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:39.081292 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:39.081294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:39.081699 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:39.081819 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:39.082512 (+ 693us) server_negotiation.cc:299] Negotiation successful
0620 19:12:39.082796 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":110,"threads_started":1}
I20260620 19:12:39.083354 25123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:39.069172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.66:34855 (local address 127.9.155.65:50065)
0620 19:12:39.069339 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:39.069356 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:39.069477 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:39.071657 (+ 2180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:39.071663 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:39.072130 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:39.072672 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:39.072689 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:39.073950 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:39.073953 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:39.074936 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:39.074945 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:39.075622 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:39.075626 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:39.075840 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:39.076501 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:12:39.076517 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:39.078505 (+ 1988us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:39.080474 (+ 1969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:39.080478 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:39.080480 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:39.080750 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:39.081059 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:39.081062 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:39.081064 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:39.081205 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:39.081811 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:39.081818 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:39.082046 (+ 228us) client_negotiation.cc:769] Sending connection context
0620 19:12:39.082216 (+ 170us) client_negotiation.cc:240] Negotiation successful
0620 19:12:39.083201 (+ 985us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71,"mutex_wait_us":843,"spinlock_wait_cycles":14976}
I20260620 19:12:39.082774 25128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:39.069512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.65:47359 (local address 127.9.155.67:44459)
0620 19:12:39.069894 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:39.069904 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:39.069933 (+ 29us) server_negotiation.cc:407] Connection header received
0620 19:12:39.070730 (+ 797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:39.070739 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:39.071201 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:39.071532 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:39.072841 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:39.073802 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:39.075050 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:39.075502 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:39.075656 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:39.078459 (+ 2803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:39.078464 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:39.078477 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:39.078507 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:39.080260 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:39.080866 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:39.080872 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:39.080876 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:39.080984 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:39.081295 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:39.081298 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:39.081300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:39.081704 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:39.081819 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:39.082343 (+ 524us) server_negotiation.cc:299] Negotiation successful
0620 19:12:39.082619 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":93,"threads_started":1}
W20260620 19:12:39.085958 24741 leader_election.cc:341] T 8ea5756386f643ffb365c87e6b5e8944 P 2b13fa8f898b4c99b52c87c89708aa37 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f70693a176b0410cac470f692003061a (127.9.155.66:34855): Not found: Tablet not found: 8ea5756386f643ffb365c87e6b5e8944
W20260620 19:12:39.086215 24742 leader_election.cc:341] T 8ea5756386f643ffb365c87e6b5e8944 P 2b13fa8f898b4c99b52c87c89708aa37 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 037d4c71bd604f338452e010c6fd92f2 (127.9.155.67:44459): Not found: Tablet not found: 8ea5756386f643ffb365c87e6b5e8944
I20260620 19:12:39.086306 24742 leader_election.cc:302] T 8ea5756386f643ffb365c87e6b5e8944 P 2b13fa8f898b4c99b52c87c89708aa37 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b13fa8f898b4c99b52c87c89708aa37; no voters: 037d4c71bd604f338452e010c6fd92f2, f70693a176b0410cac470f692003061a
I20260620 19:12:39.086602 25125 raft_consensus.cc:2726] T 8ea5756386f643ffb365c87e6b5e8944 P 2b13fa8f898b4c99b52c87c89708aa37 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:12:39.110423 25129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:39.109989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:33551 (local address 127.9.155.67:48699)
0620 19:12:39.110256 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:39.110319 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":85,"threads_started":1}
I20260620 19:12:39.801806 25130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:39.801337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:33551 (local address 127.9.155.65:34469)
0620 19:12:39.801635 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:39.801700 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":81,"threads_started":1}
I20260620 19:12:39.928393 25131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:39.927940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:33551 (local address 127.9.155.66:60873)
0620 19:12:39.928194 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:39.928301 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":71,"threads_started":1}
I20260620 19:12:40.111697 25132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:40.111188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:33551 (local address 127.9.155.67:53433)
0620 19:12:40.111509 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:40.111577 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":98,"threads_started":1}
I20260620 19:12:40.709663 25133 raft_consensus.cc:482] T 8ea5756386f643ffb365c87e6b5e8944 P 2b13fa8f898b4c99b52c87c89708aa37 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:12:40.709856 25133 raft_consensus.cc:504] T 8ea5756386f643ffb365c87e6b5e8944 P 2b13fa8f898b4c99b52c87c89708aa37 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f70693a176b0410cac470f692003061a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 34855 } } peers { permanent_uuid: "037d4c71bd604f338452e010c6fd92f2" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 44459 } } peers { permanent_uuid: "2b13fa8f898b4c99b52c87c89708aa37" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 46309 } }
I20260620 19:12:40.710351 25133 leader_election.cc:288] T 8ea5756386f643ffb365c87e6b5e8944 P 2b13fa8f898b4c99b52c87c89708aa37 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f70693a176b0410cac470f692003061a (127.9.155.66:34855), 037d4c71bd604f338452e010c6fd92f2 (127.9.155.67:44459)
W20260620 19:12:40.711263 24742 leader_election.cc:341] T 8ea5756386f643ffb365c87e6b5e8944 P 2b13fa8f898b4c99b52c87c89708aa37 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 037d4c71bd604f338452e010c6fd92f2 (127.9.155.67:44459): Not found: Tablet not found: 8ea5756386f643ffb365c87e6b5e8944
W20260620 19:12:40.711388 24741 leader_election.cc:341] T 8ea5756386f643ffb365c87e6b5e8944 P 2b13fa8f898b4c99b52c87c89708aa37 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f70693a176b0410cac470f692003061a (127.9.155.66:34855): Not found: Tablet not found: 8ea5756386f643ffb365c87e6b5e8944
I20260620 19:12:40.711475 24741 leader_election.cc:302] T 8ea5756386f643ffb365c87e6b5e8944 P 2b13fa8f898b4c99b52c87c89708aa37 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b13fa8f898b4c99b52c87c89708aa37; no voters: 037d4c71bd604f338452e010c6fd92f2, f70693a176b0410cac470f692003061a
I20260620 19:12:40.711632 25133 raft_consensus.cc:2726] T 8ea5756386f643ffb365c87e6b5e8944 P 2b13fa8f898b4c99b52c87c89708aa37 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:12:40.803128 25134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:40.802640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:33551 (local address 127.9.155.65:35205)
0620 19:12:40.802922 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:40.802984 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":93,"threads_started":1}
I20260620 19:12:40.929561 25135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:40.929091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:33551 (local address 127.9.155.66:47983)
0620 19:12:40.929379 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:40.929435 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":87,"threads_started":1}
I20260620 19:12:41.113044 25136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:41.112548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:33551 (local address 127.9.155.67:49477)
0620 19:12:41.112858 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:41.112924 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":94,"threads_started":1}
I20260620 19:12:41.804458 25137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:41.803922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:33551 (local address 127.9.155.65:44695)
0620 19:12:41.804206 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:41.804348 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":79,"threads_started":1}
I20260620 19:12:41.930902 25138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:41.930406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:33551 (local address 127.9.155.66:36423)
0620 19:12:41.930726 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:41.930799 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":87,"threads_started":1}
I20260620 19:12:42.114280 25139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:42.113834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:33551 (local address 127.9.155.67:40299)
0620 19:12:42.114108 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:42.114170 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":78,"threads_started":1}
I20260620 19:12:42.805943 25140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:42.805478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:33551 (local address 127.9.155.65:58915)
0620 19:12:42.805793 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:42.805847 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":71,"threads_started":1}
I20260620 19:12:42.932278 25141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:42.931763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:33551 (local address 127.9.155.66:49455)
0620 19:12:42.932043 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:42.932107 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":84,"threads_started":1}
I20260620 19:12:43.115533 25142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:43.115055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:33551 (local address 127.9.155.67:49335)
0620 19:12:43.115368 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:43.115432 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":99,"threads_started":1}
I20260620 19:12:43.434670 25143 raft_consensus.cc:482] T 8ea5756386f643ffb365c87e6b5e8944 P 2b13fa8f898b4c99b52c87c89708aa37 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:12:43.434825 25143 raft_consensus.cc:504] T 8ea5756386f643ffb365c87e6b5e8944 P 2b13fa8f898b4c99b52c87c89708aa37 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f70693a176b0410cac470f692003061a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 34855 } } peers { permanent_uuid: "037d4c71bd604f338452e010c6fd92f2" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 44459 } } peers { permanent_uuid: "2b13fa8f898b4c99b52c87c89708aa37" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 46309 } }
I20260620 19:12:43.435300 25143 leader_election.cc:288] T 8ea5756386f643ffb365c87e6b5e8944 P 2b13fa8f898b4c99b52c87c89708aa37 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f70693a176b0410cac470f692003061a (127.9.155.66:34855), 037d4c71bd604f338452e010c6fd92f2 (127.9.155.67:44459)
W20260620 19:12:43.436048 24741 leader_election.cc:341] T 8ea5756386f643ffb365c87e6b5e8944 P 2b13fa8f898b4c99b52c87c89708aa37 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f70693a176b0410cac470f692003061a (127.9.155.66:34855): Not found: Tablet not found: 8ea5756386f643ffb365c87e6b5e8944
W20260620 19:12:43.436287 24742 leader_election.cc:341] T 8ea5756386f643ffb365c87e6b5e8944 P 2b13fa8f898b4c99b52c87c89708aa37 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 037d4c71bd604f338452e010c6fd92f2 (127.9.155.67:44459): Not found: Tablet not found: 8ea5756386f643ffb365c87e6b5e8944
I20260620 19:12:43.436339 24742 leader_election.cc:302] T 8ea5756386f643ffb365c87e6b5e8944 P 2b13fa8f898b4c99b52c87c89708aa37 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b13fa8f898b4c99b52c87c89708aa37; no voters: 037d4c71bd604f338452e010c6fd92f2, f70693a176b0410cac470f692003061a
I20260620 19:12:43.436480 25143 raft_consensus.cc:2726] T 8ea5756386f643ffb365c87e6b5e8944 P 2b13fa8f898b4c99b52c87c89708aa37 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:12:43.807123 25144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:43.806653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:33551 (local address 127.9.155.65:54515)
0620 19:12:43.806949 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:43.807014 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":85,"threads_started":1}
I20260620 19:12:43.933430 25145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:43.933005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:33551 (local address 127.9.155.66:57869)
0620 19:12:43.933274 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:43.933332 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":74,"threads_started":1}
I20260620 19:12:44.116864 25146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:44.116370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:33551 (local address 127.9.155.67:57437)
0620 19:12:44.116701 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:44.116761 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":74,"threads_started":1}
I20260620 19:12:44.808538 25147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:44.808015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:33551 (local address 127.9.155.65:44445)
0620 19:12:44.808342 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:44.808419 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":75,"threads_started":1}
I20260620 19:12:44.934825 25148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:44.934302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:33551 (local address 127.9.155.66:55591)
0620 19:12:44.934661 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:44.934727 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":74,"threads_started":1}
I20260620 19:12:45.118103 25149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:45.117598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:33551 (local address 127.9.155.67:47773)
0620 19:12:45.117952 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:45.118005 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":123,"threads_started":1}
I20260620 19:12:45.385342 25150 raft_consensus.cc:482] T 8ea5756386f643ffb365c87e6b5e8944 P 2b13fa8f898b4c99b52c87c89708aa37 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:12:45.385486 25150 raft_consensus.cc:504] T 8ea5756386f643ffb365c87e6b5e8944 P 2b13fa8f898b4c99b52c87c89708aa37 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f70693a176b0410cac470f692003061a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 34855 } } peers { permanent_uuid: "037d4c71bd604f338452e010c6fd92f2" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 44459 } } peers { permanent_uuid: "2b13fa8f898b4c99b52c87c89708aa37" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 46309 } }
I20260620 19:12:45.385910 25150 leader_election.cc:288] T 8ea5756386f643ffb365c87e6b5e8944 P 2b13fa8f898b4c99b52c87c89708aa37 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f70693a176b0410cac470f692003061a (127.9.155.66:34855), 037d4c71bd604f338452e010c6fd92f2 (127.9.155.67:44459)
W20260620 19:12:45.386715 24742 leader_election.cc:341] T 8ea5756386f643ffb365c87e6b5e8944 P 2b13fa8f898b4c99b52c87c89708aa37 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 037d4c71bd604f338452e010c6fd92f2 (127.9.155.67:44459): Not found: Tablet not found: 8ea5756386f643ffb365c87e6b5e8944
W20260620 19:12:45.386819 24741 leader_election.cc:341] T 8ea5756386f643ffb365c87e6b5e8944 P 2b13fa8f898b4c99b52c87c89708aa37 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f70693a176b0410cac470f692003061a (127.9.155.66:34855): Not found: Tablet not found: 8ea5756386f643ffb365c87e6b5e8944
I20260620 19:12:45.386879 24741 leader_election.cc:302] T 8ea5756386f643ffb365c87e6b5e8944 P 2b13fa8f898b4c99b52c87c89708aa37 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b13fa8f898b4c99b52c87c89708aa37; no voters: 037d4c71bd604f338452e010c6fd92f2, f70693a176b0410cac470f692003061a
I20260620 19:12:45.387013 25150 raft_consensus.cc:2726] T 8ea5756386f643ffb365c87e6b5e8944 P 2b13fa8f898b4c99b52c87c89708aa37 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:12:45.809778 25151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:45.809312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:33551 (local address 127.9.155.65:53881)
0620 19:12:45.809582 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:45.809643 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":73,"threads_started":1}
I20260620 19:12:45.936278 25152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:45.935762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:33551 (local address 127.9.155.66:38371)
0620 19:12:45.936055 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:45.936137 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":78,"threads_started":1}
I20260620 19:12:46.119508 25153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:46.119046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:33551 (local address 127.9.155.67:37279)
0620 19:12:46.119331 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:46.119398 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":73,"threads_started":1}
I20260620 19:12:46.811045 25154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:46.810559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:33551 (local address 127.9.155.65:60941)
0620 19:12:46.810852 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:46.810926 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":91,"threads_started":1}
I20260620 19:12:46.937515 25155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:46.937063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:33551 (local address 127.9.155.66:60399)
0620 19:12:46.937353 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:46.937416 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":85,"threads_started":1}
I20260620 19:12:47.120747 25156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:47.120307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:33551 (local address 127.9.155.67:51579)
0620 19:12:47.120580 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:47.120644 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":73,"threads_started":1}
I20260620 19:12:47.812268 25157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:47.811826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:33551 (local address 127.9.155.65:39139)
0620 19:12:47.812062 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:47.812123 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":72,"threads_started":1}
I20260620 19:12:47.938828 25158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:47.938370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:33551 (local address 127.9.155.66:55229)
0620 19:12:47.938653 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:47.938723 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":72,"threads_started":1}
I20260620 19:12:48.122066 25159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:48.121589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:33551 (local address 127.9.155.67:43203)
0620 19:12:48.121903 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:48.121964 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":75,"threads_started":1}
I20260620 19:12:48.813360 25160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:48.812947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:33551 (local address 127.9.155.65:50057)
0620 19:12:48.813208 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:48.813264 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":69,"threads_started":1}
I20260620 19:12:48.940054 25161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:48.939679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:33551 (local address 127.9.155.66:50259)
0620 19:12:48.939908 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:48.939963 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":70,"threads_started":1}
I20260620 19:12:49.123449 25162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:49.122977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:33551 (local address 127.9.155.67:46163)
0620 19:12:49.123277 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:49.123344 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":89,"threads_started":1}
I20260620 19:12:49.814584 25163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:49.814148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:33551 (local address 127.9.155.65:35761)
0620 19:12:49.814426 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:49.814487 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":77,"threads_started":1}
I20260620 19:12:49.941313 25164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:49.940864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:33551 (local address 127.9.155.66:56171)
0620 19:12:49.941137 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:49.941204 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":85,"threads_started":1}
I20260620 19:12:50.124871 25165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:50.124355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:33551 (local address 127.9.155.67:43107)
0620 19:12:50.124706 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:50.124771 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":122,"threads_started":1}
I20260620 19:12:50.815929 25166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:50.815468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:33551 (local address 127.9.155.65:58527)
0620 19:12:50.815765 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:50.815830 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":75,"threads_started":1}
I20260620 19:12:50.942538 25167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:50.942120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:33551 (local address 127.9.155.66:56293)
0620 19:12:50.942387 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:50.942442 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":90,"threads_started":1}
I20260620 19:12:51.126241 25168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:51.125790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:33551 (local address 127.9.155.67:33567)
0620 19:12:51.126077 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:51.126140 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":88,"threads_started":1}
I20260620 19:12:51.750802 25169 raft_consensus.cc:482] T 8ea5756386f643ffb365c87e6b5e8944 P 2b13fa8f898b4c99b52c87c89708aa37 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:12:51.750947 25169 raft_consensus.cc:504] T 8ea5756386f643ffb365c87e6b5e8944 P 2b13fa8f898b4c99b52c87c89708aa37 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f70693a176b0410cac470f692003061a" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 34855 } } peers { permanent_uuid: "037d4c71bd604f338452e010c6fd92f2" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 44459 } } peers { permanent_uuid: "2b13fa8f898b4c99b52c87c89708aa37" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 46309 } }
I20260620 19:12:51.751348 25169 leader_election.cc:288] T 8ea5756386f643ffb365c87e6b5e8944 P 2b13fa8f898b4c99b52c87c89708aa37 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f70693a176b0410cac470f692003061a (127.9.155.66:34855), 037d4c71bd604f338452e010c6fd92f2 (127.9.155.67:44459)
W20260620 19:12:51.752059 24741 leader_election.cc:341] T 8ea5756386f643ffb365c87e6b5e8944 P 2b13fa8f898b4c99b52c87c89708aa37 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f70693a176b0410cac470f692003061a (127.9.155.66:34855): Not found: Tablet not found: 8ea5756386f643ffb365c87e6b5e8944
W20260620 19:12:51.752168 24742 leader_election.cc:341] T 8ea5756386f643ffb365c87e6b5e8944 P 2b13fa8f898b4c99b52c87c89708aa37 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 037d4c71bd604f338452e010c6fd92f2 (127.9.155.67:44459): Not found: Tablet not found: 8ea5756386f643ffb365c87e6b5e8944
I20260620 19:12:51.752266 24742 leader_election.cc:302] T 8ea5756386f643ffb365c87e6b5e8944 P 2b13fa8f898b4c99b52c87c89708aa37 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b13fa8f898b4c99b52c87c89708aa37; no voters: 037d4c71bd604f338452e010c6fd92f2, f70693a176b0410cac470f692003061a
I20260620 19:12:51.752413 25169 raft_consensus.cc:2726] T 8ea5756386f643ffb365c87e6b5e8944 P 2b13fa8f898b4c99b52c87c89708aa37 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:12:51.817122 25170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:51.816696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:33551 (local address 127.9.155.65:34079)
0620 19:12:51.816963 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:51.817018 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":75,"threads_started":1}
I20260620 19:12:51.943823 25171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:51.943352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:33551 (local address 127.9.155.66:56491)
0620 19:12:51.943661 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:51.943726 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":79,"threads_started":1}
I20260620 19:12:52.127538 25172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:52.127071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:33551 (local address 127.9.155.67:58429)
0620 19:12:52.127345 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:52.127410 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":89,"threads_started":1}
I20260620 19:12:52.818404 25173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:52.818005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:33551 (local address 127.9.155.65:38573)
0620 19:12:52.818256 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:52.818311 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":88,"threads_started":1}
I20260620 19:12:52.945252 25174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:52.944838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:33551 (local address 127.9.155.66:48083)
0620 19:12:52.945091 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:52.945154 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":73,"threads_started":1}
I20260620 19:12:53.121588 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 24689
I20260620 19:12:53.127308 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 24856
I20260620 19:12:53.129012 25175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:53.128480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:33551 (local address 127.9.155.67:55403)
0620 19:12:53.128848 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:53.128909 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":81,"threads_started":1}
I20260620 19:12:53.132248 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 24990
2026-06-20T19:12:53Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19161 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:12:53 dist-test-slave-n3fx krb5kdc[25178](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:12:53 dist-test-slave-n3fx krb5kdc[25178](info): set up 2 sockets
Jun 20 19:12:53 dist-test-slave-n3fx krb5kdc[25178](info): commencing operation
krb5kdc: starting...
W20260620 19:12:54.108979 9837 mini_kdc.cc:121] Time spent starting KDC: real 0.956s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/krb5kdc/test-user.keytab.
Jun 20 19:12:54 dist-test-slave-n3fx krb5kdc[25178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982774, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:12:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:12:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.155.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.155.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.155.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab.
Entry for principal kudu/127.9.155.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab.
I20260620 19:12:54.228032 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:40693
--webserver_interface=127.9.155.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab
--principal=kudu/127.9.155.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:42519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.155.126:40693
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:12:54.341315 25191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:54.341600 25191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:54.341648 25191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:54.345717 25191 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 19:12:54.345781 25191 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 19:12:54.345796 25191 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 19:12:54.345808 25191 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:12:54.345825 25191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:54.345842 25191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:54.345870 25191 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:12:54.345885 25191 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:12:54.351444 25191 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:42519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.9.155.126:40693
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab
--principal=kudu/127.9.155.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:40693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.155.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:12:54.352550 25191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:54.353554 25191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 19:12:54.362377 25191 server_base.cc:1034] running on GCE node
W20260620 19:12:54.362466 25196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:54.362464 25199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:54.362794 25197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:54.363440 25191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:54.364651 25191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:54.365805 25191 hybrid_clock.cc:648] HybridClock initialized: now 1781982774365791 us; error 30 us; skew 500 ppm
Jun 20 19:12:54 dist-test-slave-n3fx krb5kdc[25178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982774, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:54.368472 25191 init.cc:377] Logged in from keytab as kudu/127.9.155.126@KRBTEST.COM (short username kudu)
I20260620 19:12:54.369333 25191 webserver.cc:460] Webserver started at http://127.9.155.126:40719/ using document root <none> and password file <none>
I20260620 19:12:54.369853 25191 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:54.369915 25191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:12:54.370113 25191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:12:54.372270 25191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/master-0/data/instance:
uuid: "025239f00c2441fe885affb9769996d9"
format_stamp: "Formatted at 2026-06-20 19:12:54 on dist-test-slave-n3fx"
server_key: "0c722aadc3be79eeb06f6000b03a0746"
server_key_iv: "a9c7db7ad71d08c9f106596a6f7c0434"
server_key_version: "encryptionkey@0"
I20260620 19:12:54.372851 25191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/master-0/wal/instance:
uuid: "025239f00c2441fe885affb9769996d9"
format_stamp: "Formatted at 2026-06-20 19:12:54 on dist-test-slave-n3fx"
server_key: "0c722aadc3be79eeb06f6000b03a0746"
server_key_iv: "a9c7db7ad71d08c9f106596a6f7c0434"
server_key_version: "encryptionkey@0"
I20260620 19:12:54.376886 25191 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 19:12:54.379294 25206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:54.380620 25191 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:12:54.380784 25191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/master-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/master-0/wal
uuid: "025239f00c2441fe885affb9769996d9"
format_stamp: "Formatted at 2026-06-20 19:12:54 on dist-test-slave-n3fx"
server_key: "0c722aadc3be79eeb06f6000b03a0746"
server_key_iv: "a9c7db7ad71d08c9f106596a6f7c0434"
server_key_version: "encryptionkey@0"
I20260620 19:12:54.380890 25191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:54.396067 25191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:54.399255 25191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:54.399436 25191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:54.406354 25191 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.126:40693
I20260620 19:12:54.406738 25258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.126:40693 every 8 connection(s)
I20260620 19:12:54.407359 25191 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
I20260620 19:12:54.410157 25259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:12:54.414178 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 25191
I20260620 19:12:54.414295 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/master-0/wal/instance
I20260620 19:12:54.414551 9837 external_mini_cluster.cc:1428] Setting key 26580087e99453c49a454a2a9a102d6c
I20260620 19:12:54.416087 25259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 025239f00c2441fe885affb9769996d9: Bootstrap starting.
I20260620 19:12:54.418387 25259 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 025239f00c2441fe885affb9769996d9: Neither blocks nor log segments found. Creating new log.
I20260620 19:12:54.419276 25259 log.cc:824] T 00000000000000000000000000000000 P 025239f00c2441fe885affb9769996d9: Log is configured to *not* fsync() on all Append() calls
I20260620 19:12:54.421519 25259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 025239f00c2441fe885affb9769996d9: No bootstrap required, opened a new log
Jun 20 19:12:54 dist-test-slave-n3fx krb5kdc[25178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982774, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:12:54.425760 25259 raft_consensus.cc:348] T 00000000000000000000000000000000 P 025239f00c2441fe885affb9769996d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "025239f00c2441fe885affb9769996d9" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 40693 } }
I20260620 19:12:54.426040 25259 raft_consensus.cc:374] T 00000000000000000000000000000000 P 025239f00c2441fe885affb9769996d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:12:54.426110 25259 raft_consensus.cc:729] T 00000000000000000000000000000000 P 025239f00c2441fe885affb9769996d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 025239f00c2441fe885affb9769996d9, State: Initialized, Role: FOLLOWER
I20260620 19:12:54.426625 25259 consensus_queue.cc:260] T 00000000000000000000000000000000 P 025239f00c2441fe885affb9769996d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "025239f00c2441fe885affb9769996d9" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 40693 } }
I20260620 19:12:54.426761 25259 raft_consensus.cc:388] T 00000000000000000000000000000000 P 025239f00c2441fe885affb9769996d9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:12:54.426856 25259 raft_consensus.cc:482] T 00000000000000000000000000000000 P 025239f00c2441fe885affb9769996d9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:12:54.426963 25259 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 025239f00c2441fe885affb9769996d9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:12:54.428009 25259 raft_consensus.cc:504] T 00000000000000000000000000000000 P 025239f00c2441fe885affb9769996d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "025239f00c2441fe885affb9769996d9" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 40693 } }
I20260620 19:12:54.428412 25259 leader_election.cc:302] T 00000000000000000000000000000000 P 025239f00c2441fe885affb9769996d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 025239f00c2441fe885affb9769996d9; no voters:
I20260620 19:12:54.428743 25259 leader_election.cc:288] T 00000000000000000000000000000000 P 025239f00c2441fe885affb9769996d9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:12:54.428836 25264 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 025239f00c2441fe885affb9769996d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:12:54.429035 25264 raft_consensus.cc:686] T 00000000000000000000000000000000 P 025239f00c2441fe885affb9769996d9 [term 1 LEADER]: Becoming Leader. State: Replica: 025239f00c2441fe885affb9769996d9, State: Running, Role: LEADER
I20260620 19:12:54.429386 25264 consensus_queue.cc:237] T 00000000000000000000000000000000 P 025239f00c2441fe885affb9769996d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "025239f00c2441fe885affb9769996d9" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 40693 } }
I20260620 19:12:54.429545 25262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:54.415943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52188 (local address 127.9.155.126:40693)
0620 19:12:54.416446 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:54.416462 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:54.416497 (+ 35us) server_negotiation.cc:407] Connection header received
0620 19:12:54.417179 (+ 682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:54.417200 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:54.417541 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:54.417864 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:54.418693 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:54.420017 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:54.421065 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:54.421752 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:54.421916 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:54.424462 (+ 2546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:54.424470 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:54.424491 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:54.424528 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:54.426381 (+ 1853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:54.426843 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:54.426849 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:54.426856 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:54.426995 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:54.427312 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:54.427315 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:54.427316 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:54.427782 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:54.427939 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:54.428306 (+ 367us) server_negotiation.cc:299] Negotiation successful
0620 19:12:54.428608 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":168,"threads_started":1}
I20260620 19:12:54.430015 25259 sys_catalog.cc:564] T 00000000000000000000000000000000 P 025239f00c2441fe885affb9769996d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:12:54.430606 25264 sys_catalog.cc:455] T 00000000000000000000000000000000 P 025239f00c2441fe885affb9769996d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 025239f00c2441fe885affb9769996d9. Latest consensus state: current_term: 1 leader_uuid: "025239f00c2441fe885affb9769996d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "025239f00c2441fe885affb9769996d9" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 40693 } } }
I20260620 19:12:54.430717 25264 sys_catalog.cc:458] T 00000000000000000000000000000000 P 025239f00c2441fe885affb9769996d9 [sys.catalog]: This master's current role is: LEADER
I20260620 19:12:54.431043 25265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 025239f00c2441fe885affb9769996d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "025239f00c2441fe885affb9769996d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "025239f00c2441fe885affb9769996d9" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 40693 } } }
I20260620 19:12:54.431141 25265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 025239f00c2441fe885affb9769996d9 [sys.catalog]: This master's current role is: LEADER
I20260620 19:12:54.431582 25268 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:12:54.434845 25268 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:12:54.440528 25268 catalog_manager.cc:1347] Generated new cluster ID: 36f418d264f64c6fb041fa4533bace46
I20260620 19:12:54.440603 25268 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:12:54.462955 25268 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:12:54.463996 25268 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:12:54.476483 25268 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 025239f00c2441fe885affb9769996d9: Generated new TSK 0
I20260620 19:12:54.477254 25268 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.155.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.155.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.155.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.65.keytab.
Entry for principal kudu/127.9.155.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.65.keytab.
I20260620 19:12:54.508109 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.65:0
--local_ip_for_outbound_sockets=127.9.155.65
--webserver_interface=127.9.155.65
--webserver_port=0
--tserver_master_addrs=127.9.155.126:40693
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.65.keytab
--principal=kudu/127.9.155.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:42519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:12:54.618587 25285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:54.618865 25285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:54.618909 25285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:54.622922 25285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:54.622994 25285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:54.623054 25285 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.65
I20260620 19:12:54.628575 25285 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:42519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.65.keytab
--principal=kudu/127.9.155.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.155.65
--webserver_port=0
--tserver_master_addrs=127.9.155.126:40693
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25285
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.65
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:12:54.629738 25285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:54.630815 25285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:12:54.640980 25290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:54.641006 25291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:54.640980 25293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:54.641191 25285 server_base.cc:1034] running on GCE node
I20260620 19:12:54.641741 25285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:54.642403 25285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:54.643612 25285 hybrid_clock.cc:648] HybridClock initialized: now 1781982774643606 us; error 71 us; skew 500 ppm
Jun 20 19:12:54 dist-test-slave-n3fx krb5kdc[25178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982774, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:54.646514 25285 init.cc:377] Logged in from keytab as kudu/127.9.155.65@KRBTEST.COM (short username kudu)
I20260620 19:12:54.647390 25285 webserver.cc:460] Webserver started at http://127.9.155.65:36173/ using document root <none> and password file <none>
I20260620 19:12:54.647914 25285 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:54.647971 25285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:12:54.648135 25285 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:12:54.650173 25285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-0/data/instance:
uuid: "2dee97c110914202b2e9ca8db83df0bf"
format_stamp: "Formatted at 2026-06-20 19:12:54 on dist-test-slave-n3fx"
server_key: "6c811a05433d6736303f0a3060afa774"
server_key_iv: "d75bd652d3fdee25dbbd0add8a9744a9"
server_key_version: "encryptionkey@0"
I20260620 19:12:54.650689 25285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-0/wal/instance:
uuid: "2dee97c110914202b2e9ca8db83df0bf"
format_stamp: "Formatted at 2026-06-20 19:12:54 on dist-test-slave-n3fx"
server_key: "6c811a05433d6736303f0a3060afa774"
server_key_iv: "d75bd652d3fdee25dbbd0add8a9744a9"
server_key_version: "encryptionkey@0"
I20260620 19:12:54.654680 25285 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 19:12:54.657100 25300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:54.658383 25285 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 19:12:54.658480 25285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-0/wal
uuid: "2dee97c110914202b2e9ca8db83df0bf"
format_stamp: "Formatted at 2026-06-20 19:12:54 on dist-test-slave-n3fx"
server_key: "6c811a05433d6736303f0a3060afa774"
server_key_iv: "d75bd652d3fdee25dbbd0add8a9744a9"
server_key_version: "encryptionkey@0"
I20260620 19:12:54.658573 25285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:54.670447 25285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:54.673244 25285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:54.673400 25285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:54.673918 25285 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:12:54.674714 25285 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:12:54.674760 25285 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:54.674799 25285 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:12:54.674819 25285 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:54.682988 25285 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.65:39169
I20260620 19:12:54.683041 25413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.65:39169 every 8 connection(s)
I20260620 19:12:54.683871 25285 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
I20260620 19:12:54.684721 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 25285
I20260620 19:12:54.684818 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-0/wal/instance
I20260620 19:12:54.685060 9837 external_mini_cluster.cc:1428] Setting key 46ab302f69174d1c1a15201a4a858d5e
WARNING: no policy specified for kudu/127.9.155.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.155.66@KRBTEST.COM" created.
Jun 20 19:12:54 dist-test-slave-n3fx krb5kdc[25178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982774, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.65@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:12:54.700771 25262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:54.685811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.65:41527 (local address 127.9.155.126:40693)
0620 19:12:54.685943 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:54.685947 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:54.688340 (+ 2393us) server_negotiation.cc:407] Connection header received
0620 19:12:54.689600 (+ 1260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:54.689604 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:54.689660 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:54.689758 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:54.691093 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:54.691941 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:54.693259 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:54.693619 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:54.693745 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:54.697327 (+ 3582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:54.697330 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:54.697332 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:54.697361 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:54.698680 (+ 1319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:54.699211 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:54.699215 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:54.699216 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:54.699294 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:54.699623 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:54.699626 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:54.699627 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:54.699823 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:54.699907 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:54.700463 (+ 556us) server_negotiation.cc:299] Negotiation successful
0620 19:12:54.700599 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 19:12:54.701560 25416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:54.686216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40693 (local address 127.9.155.65:41527)
0620 19:12:54.688158 (+ 1942us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:54.688201 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:54.689386 (+ 1185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:54.689925 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:54.689937 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:54.690300 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:54.690855 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:54.690875 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:54.692093 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:54.692096 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:54.693119 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:54.693130 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:54.693824 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:54.693829 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:54.694010 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:54.694546 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:12:54.694562 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:54.697120 (+ 2558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:54.698812 (+ 1692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:54.698818 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:54.698837 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:54.699094 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:54.699379 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:54.699382 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:54.699384 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:54.699522 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:54.699927 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:54.699934 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:54.700182 (+ 248us) client_negotiation.cc:769] Sending connection context
0620 19:12:54.700394 (+ 212us) client_negotiation.cc:240] Negotiation successful
0620 19:12:54.700746 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1752,"thread_start_us":152,"threads_started":1}
I20260620 19:12:54.702904 25414 heartbeater.cc:347] Connected to a master server at 127.9.155.126:40693
I20260620 19:12:54.703181 25414 heartbeater.cc:464] Registering TS with master...
I20260620 19:12:54.703727 25414 heartbeater.cc:511] Master 127.9.155.126:40693 requested a full tablet report, sending...
I20260620 19:12:54.705303 25223 ts_manager.cc:194] Registered new tserver with Master: 2dee97c110914202b2e9ca8db83df0bf (127.9.155.65:39169)
I20260620 19:12:54.706717 25223 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.155.65@KRBTEST.COM'} at 127.9.155.65:41527
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.155.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.66.keytab.
Entry for principal kudu/127.9.155.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.66.keytab.
I20260620 19:12:54.712507 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.66:0
--local_ip_for_outbound_sockets=127.9.155.66
--webserver_interface=127.9.155.66
--webserver_port=0
--tserver_master_addrs=127.9.155.126:40693
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.66.keytab
--principal=kudu/127.9.155.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:42519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:12:54.822279 25419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:54.822540 25419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:54.822592 25419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:54.826633 25419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:54.826711 25419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:54.826773 25419 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.66
I20260620 19:12:54.832161 25419 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:42519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.66.keytab
--principal=kudu/127.9.155.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.155.66
--webserver_port=0
--tserver_master_addrs=127.9.155.126:40693
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25419
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.66
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:12:54.833261 25419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:54.834244 25419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:12:54.844055 25424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:54.844089 25427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:54.844070 25419 server_base.cc:1034] running on GCE node
W20260620 19:12:54.844099 25425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:54.844800 25419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:54.845428 25419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:54.846582 25419 hybrid_clock.cc:648] HybridClock initialized: now 1781982774846564 us; error 31 us; skew 500 ppm
Jun 20 19:12:54 dist-test-slave-n3fx krb5kdc[25178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982774, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:54.849555 25419 init.cc:377] Logged in from keytab as kudu/127.9.155.66@KRBTEST.COM (short username kudu)
I20260620 19:12:54.850530 25419 webserver.cc:460] Webserver started at http://127.9.155.66:45103/ using document root <none> and password file <none>
I20260620 19:12:54.851111 25419 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:54.851173 25419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:12:54.851461 25419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:12:54.853782 25419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-1/data/instance:
uuid: "a830c89ba25b4f8cb3b47d173e20ff3e"
format_stamp: "Formatted at 2026-06-20 19:12:54 on dist-test-slave-n3fx"
server_key: "5443a0cb3a88e72d78fd8b772e429b51"
server_key_iv: "6d83aac77ad59a26c37f02f9f4b7f1bc"
server_key_version: "encryptionkey@0"
I20260620 19:12:54.854372 25419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-1/wal/instance:
uuid: "a830c89ba25b4f8cb3b47d173e20ff3e"
format_stamp: "Formatted at 2026-06-20 19:12:54 on dist-test-slave-n3fx"
server_key: "5443a0cb3a88e72d78fd8b772e429b51"
server_key_iv: "6d83aac77ad59a26c37f02f9f4b7f1bc"
server_key_version: "encryptionkey@0"
I20260620 19:12:54.858358 25419 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 19:12:54.860877 25434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:54.862246 25419 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:12:54.862360 25419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-1/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-1/wal
uuid: "a830c89ba25b4f8cb3b47d173e20ff3e"
format_stamp: "Formatted at 2026-06-20 19:12:54 on dist-test-slave-n3fx"
server_key: "5443a0cb3a88e72d78fd8b772e429b51"
server_key_iv: "6d83aac77ad59a26c37f02f9f4b7f1bc"
server_key_version: "encryptionkey@0"
I20260620 19:12:54.862488 25419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:54.872565 25419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:54.875576 25419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:54.875768 25419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:54.876406 25419 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:12:54.877233 25419 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:12:54.877279 25419 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:54.877331 25419 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:12:54.877360 25419 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:54.885923 25419 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.66:45875
I20260620 19:12:54.885941 25547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.66:45875 every 8 connection(s)
I20260620 19:12:54.886850 25419 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
I20260620 19:12:54.888768 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 25419
I20260620 19:12:54.888881 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-1/wal/instance
I20260620 19:12:54.889181 9837 external_mini_cluster.cc:1428] Setting key 7e698ae110a2cd0752d7a15d0468b17b
Jun 20 19:12:54 dist-test-slave-n3fx krb5kdc[25178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982774, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.66@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.155.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.155.67@KRBTEST.COM" created.
I20260620 19:12:54.902667 25262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:54.888830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.66:36531 (local address 127.9.155.126:40693)
0620 19:12:54.888971 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:54.888975 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:54.890044 (+ 1069us) server_negotiation.cc:407] Connection header received
0620 19:12:54.891585 (+ 1541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:54.891589 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:54.891643 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:54.891720 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:54.893266 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:54.893792 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:54.895518 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:54.895759 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:54.895845 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:54.898850 (+ 3005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:54.898853 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:54.898855 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:54.898882 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:54.900322 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:54.900915 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:54.900918 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:54.900919 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:54.900998 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:54.901449 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:54.901452 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:54.901454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:54.901704 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:54.901811 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:54.902365 (+ 554us) server_negotiation.cc:299] Negotiation successful
0620 19:12:54.902500 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 19:12:54.903749 25550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:54.889308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40693 (local address 127.9.155.66:36531)
0620 19:12:54.889900 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:54.889945 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:54.891360 (+ 1415us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:54.891914 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:54.891921 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:54.892410 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:54.893094 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:54.893115 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:54.893920 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:54.893923 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:54.895300 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:54.895313 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:54.895934 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:54.895938 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:54.896150 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:54.896825 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:12:54.896847 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:54.898638 (+ 1791us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:54.900434 (+ 1796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:54.900439 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:54.900454 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:54.900825 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:54.901110 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:54.901113 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:54.901116 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:54.901317 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:54.901825 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:54.901833 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:54.902108 (+ 275us) client_negotiation.cc:769] Sending connection context
0620 19:12:54.902290 (+ 182us) client_negotiation.cc:240] Negotiation successful
0620 19:12:54.902671 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":156,"threads_started":1}
I20260620 19:12:54.905287 25548 heartbeater.cc:347] Connected to a master server at 127.9.155.126:40693
I20260620 19:12:54.905541 25548 heartbeater.cc:464] Registering TS with master...
I20260620 19:12:54.906041 25548 heartbeater.cc:511] Master 127.9.155.126:40693 requested a full tablet report, sending...
I20260620 19:12:54.907231 25223 ts_manager.cc:194] Registered new tserver with Master: a830c89ba25b4f8cb3b47d173e20ff3e (127.9.155.66:45875)
I20260620 19:12:54.908031 25223 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.155.66@KRBTEST.COM'} at 127.9.155.66:36531
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.155.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.67.keytab.
Entry for principal kudu/127.9.155.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.67.keytab.
I20260620 19:12:54.919149 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.67:0
--local_ip_for_outbound_sockets=127.9.155.67
--webserver_interface=127.9.155.67
--webserver_port=0
--tserver_master_addrs=127.9.155.126:40693
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.67.keytab
--principal=kudu/127.9.155.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:42519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:12:55.029294 25553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:55.029666 25553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:55.029731 25553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:55.033979 25553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:55.034049 25553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:55.034111 25553 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.67
I20260620 19:12:55.039382 25553 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:42519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.67.keytab
--principal=kudu/127.9.155.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.155.67
--webserver_port=0
--tserver_master_addrs=127.9.155.126:40693
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25553
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.67
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:12:55.040488 25553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:55.041457 25553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:12:55.051020 25558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:55.051030 25561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:55.051020 25559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:55.051465 25553 server_base.cc:1034] running on GCE node
I20260620 19:12:55.051894 25553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:55.052620 25553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:55.053808 25553 hybrid_clock.cc:648] HybridClock initialized: now 1781982775053784 us; error 39 us; skew 500 ppm
Jun 20 19:12:55 dist-test-slave-n3fx krb5kdc[25178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982775, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:55.057099 25553 init.cc:377] Logged in from keytab as kudu/127.9.155.67@KRBTEST.COM (short username kudu)
I20260620 19:12:55.058161 25553 webserver.cc:460] Webserver started at http://127.9.155.67:45161/ using document root <none> and password file <none>
I20260620 19:12:55.058748 25553 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:55.058821 25553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:12:55.059020 25553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:12:55.061190 25553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-2/data/instance:
uuid: "0b46980251d54a8bb398e0da234a85ec"
format_stamp: "Formatted at 2026-06-20 19:12:55 on dist-test-slave-n3fx"
server_key: "d1c948a36f6cce92e06df2322361459e"
server_key_iv: "4557675b576f09102bea8534fc65fb9e"
server_key_version: "encryptionkey@0"
I20260620 19:12:55.061769 25553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-2/wal/instance:
uuid: "0b46980251d54a8bb398e0da234a85ec"
format_stamp: "Formatted at 2026-06-20 19:12:55 on dist-test-slave-n3fx"
server_key: "d1c948a36f6cce92e06df2322361459e"
server_key_iv: "4557675b576f09102bea8534fc65fb9e"
server_key_version: "encryptionkey@0"
I20260620 19:12:55.065718 25553 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 19:12:55.068322 25568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:55.069883 25553 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:12:55.070015 25553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-2/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-2/wal
uuid: "0b46980251d54a8bb398e0da234a85ec"
format_stamp: "Formatted at 2026-06-20 19:12:55 on dist-test-slave-n3fx"
server_key: "d1c948a36f6cce92e06df2322361459e"
server_key_iv: "4557675b576f09102bea8534fc65fb9e"
server_key_version: "encryptionkey@0"
I20260620 19:12:55.070148 25553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:55.085500 25553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:55.088907 25553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:55.089097 25553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:55.089706 25553 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:12:55.090544 25553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:12:55.090600 25553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:55.090655 25553 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:12:55.090692 25553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:55.099295 25553 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.67:38157
I20260620 19:12:55.099323 25681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.67:38157 every 8 connection(s)
I20260620 19:12:55.100178 25553 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
I20260620 19:12:55.105528 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 25553
I20260620 19:12:55.105649 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-2/wal/instance
I20260620 19:12:55.105897 9837 external_mini_cluster.cc:1428] Setting key fbe362894546e4b8ca47d818094b6fb4
Jun 20 19:12:55 dist-test-slave-n3fx krb5kdc[25178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982775, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.67@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:12:55.114456 25262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:55.101996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.67:36829 (local address 127.9.155.126:40693)
0620 19:12:55.102175 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:55.102186 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:55.102987 (+ 801us) server_negotiation.cc:407] Connection header received
0620 19:12:55.104290 (+ 1303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:55.104294 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:55.104348 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:55.104425 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:55.105754 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:55.106339 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:55.107762 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:55.108079 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:55.108171 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:55.110714 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:55.110718 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:55.110720 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:55.110748 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:55.112199 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:55.112841 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:55.112845 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:55.112846 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:55.112927 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:55.113308 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:55.113311 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:55.113312 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:55.113509 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:55.113586 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:55.114207 (+ 621us) server_negotiation.cc:299] Negotiation successful
0620 19:12:55.114329 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 19:12:55.115350 25684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:55.102375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40693 (local address 127.9.155.67:36829)
0620 19:12:55.102829 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:55.102871 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:55.104027 (+ 1156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:55.104583 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:55.104590 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:55.104978 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:55.105562 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:55.105584 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:55.106466 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:55.106469 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:55.107622 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:55.107631 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:55.108256 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:55.108261 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:55.108418 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:55.108954 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:12:55.108971 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:55.110497 (+ 1526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:55.112378 (+ 1881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:55.112383 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:55.112398 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:55.112741 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:55.113035 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:55.113038 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:55.113040 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:55.113209 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:55.113664 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:55.113671 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:55.113946 (+ 275us) client_negotiation.cc:769] Sending connection context
0620 19:12:55.114144 (+ 198us) client_negotiation.cc:240] Negotiation successful
0620 19:12:55.114434 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":118,"threads_started":1}
I20260620 19:12:55.116818 25682 heartbeater.cc:347] Connected to a master server at 127.9.155.126:40693
I20260620 19:12:55.117107 25682 heartbeater.cc:464] Registering TS with master...
I20260620 19:12:55.117659 25682 heartbeater.cc:511] Master 127.9.155.126:40693 requested a full tablet report, sending...
I20260620 19:12:55.118954 25223 ts_manager.cc:194] Registered new tserver with Master: 0b46980251d54a8bb398e0da234a85ec (127.9.155.67:38157)
I20260620 19:12:55.119660 25223 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.155.67@KRBTEST.COM'} at 127.9.155.67:36829
I20260620 19:12:55.119884 9837 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:12:55.129112 25262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:55.122141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52196 (local address 127.9.155.126:40693)
0620 19:12:55.122289 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:55.122294 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:55.122422 (+ 128us) server_negotiation.cc:407] Connection header received
0620 19:12:55.122530 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:55.122533 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:55.122582 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:55.122651 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:55.123320 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:55.123927 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:55.124822 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:55.125031 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:55.125111 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:55.126334 (+ 1223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:55.126337 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:55.126338 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:55.126362 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:55.127630 (+ 1268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:55.128020 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:55.128023 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:55.128024 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:55.128102 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:55.128369 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:55.128372 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:55.128373 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:55.128585 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:55.128726 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:55.128865 (+ 139us) server_negotiation.cc:299] Negotiation successful
0620 19:12:55.128956 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 19:12:55.131851 25223 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52196:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:12:55.135401 25223 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:12:55.149502 25696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:55.143636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.67:38157 (local address 127.0.0.1:54682)
0620 19:12:55.144811 (+ 1175us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:55.144990 (+ 179us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:55.145139 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:55.145642 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:55.145647 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:55.145669 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:12:55.145804 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:55.145811 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:55.146889 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:55.146892 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:55.148092 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:55.148101 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:55.148950 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:55.148953 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:55.149063 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:55.149105 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 19:12:55.149217 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 19:12:55.149342 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1006,"mutex_wait_us":1,"spinlock_wait_cycles":19456,"thread_start_us":132,"threads_started":1}
I20260620 19:12:55.149503 25695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:55.143933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54682 (local address 127.9.155.67:38157)
0620 19:12:55.144446 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:55.144458 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:55.145072 (+ 614us) server_negotiation.cc:407] Connection header received
0620 19:12:55.145333 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:55.145339 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:55.145492 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:55.145624 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:12:55.145927 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:55.146760 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:55.148241 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:55.148832 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:55.148939 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:55.149090 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:55.149254 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 19:12:55.149353 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":130,"threads_started":1}
I20260620 19:12:55.150300 25694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:55.143734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.65:39169 (local address 127.0.0.1:59102)
0620 19:12:55.144939 (+ 1205us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:55.144989 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:55.145139 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:55.145813 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:55.145815 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:55.145837 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:12:55.145965 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:55.145972 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:55.147432 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:55.147435 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:55.148694 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:55.148705 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:55.149921 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:55.149924 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:55.149999 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:55.150015 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 19:12:55.150080 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 19:12:55.150150 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1151,"mutex_wait_us":41,"spinlock_wait_cycles":30848,"thread_start_us":111,"threads_started":1}
I20260620 19:12:55.150616 25697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:55.144349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59102 (local address 127.9.155.65:39169)
0620 19:12:55.144598 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:55.144608 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:55.145125 (+ 517us) server_negotiation.cc:407] Connection header received
0620 19:12:55.145296 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:55.145302 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:55.145502 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:55.145640 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:12:55.146127 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:55.147290 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:55.148832 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:55.149600 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:55.149719 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:55.149860 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:55.150132 (+ 272us) server_negotiation.cc:299] Negotiation successful
0620 19:12:55.150234 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":95,"threads_started":1}
I20260620 19:12:55.150913 25698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:55.144480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47738 (local address 127.9.155.66:45875)
0620 19:12:55.144842 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:55.144854 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:55.145275 (+ 421us) server_negotiation.cc:407] Connection header received
0620 19:12:55.145344 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:55.145348 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:55.145502 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:55.145640 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:12:55.146057 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:55.147433 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:55.148854 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:55.149652 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:55.150001 (+ 349us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:55.150135 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:55.150611 (+ 476us) server_negotiation.cc:299] Negotiation successful
0620 19:12:55.150777 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":149,"threads_started":1}
I20260620 19:12:55.151261 25693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:55.143996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.66:45875 (local address 127.0.0.1:47738)
0620 19:12:55.144996 (+ 1000us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:55.145009 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:55.145140 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:55.145808 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:55.145810 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:55.145837 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:12:55.145942 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:55.145948 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:55.147562 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:55.147566 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:55.148735 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:55.148742 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:55.150206 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:55.150209 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:55.150297 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:55.150319 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 19:12:55.151046 (+ 727us) client_negotiation.cc:240] Negotiation successful
0620 19:12:55.151129 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":944,"mutex_wait_us":31,"spinlock_wait_cycles":15360,"thread_start_us":83,"threads_started":1}
I20260620 19:12:55.153231 25348 tablet_service.cc:1467] Processing CreateTablet for tablet 6ab3ae920c554934a7b36da4f8ee4a66 (DEFAULT_TABLE table=test-table [id=2a607240899b4b9e993b7a383a4f52ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:12:55.153120 25616 tablet_service.cc:1467] Processing CreateTablet for tablet 6ab3ae920c554934a7b36da4f8ee4a66 (DEFAULT_TABLE table=test-table [id=2a607240899b4b9e993b7a383a4f52ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:12:55.154145 25348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ab3ae920c554934a7b36da4f8ee4a66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:12:55.154179 25616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ab3ae920c554934a7b36da4f8ee4a66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:12:55.154023 25482 tablet_service.cc:1467] Processing CreateTablet for tablet 6ab3ae920c554934a7b36da4f8ee4a66 (DEFAULT_TABLE table=test-table [id=2a607240899b4b9e993b7a383a4f52ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:12:55.155021 25482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ab3ae920c554934a7b36da4f8ee4a66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:12:55.160578 25699 tablet_bootstrap.cc:492] T 6ab3ae920c554934a7b36da4f8ee4a66 P 2dee97c110914202b2e9ca8db83df0bf: Bootstrap starting.
I20260620 19:12:55.161674 25700 tablet_bootstrap.cc:492] T 6ab3ae920c554934a7b36da4f8ee4a66 P 0b46980251d54a8bb398e0da234a85ec: Bootstrap starting.
I20260620 19:12:55.162473 25701 tablet_bootstrap.cc:492] T 6ab3ae920c554934a7b36da4f8ee4a66 P a830c89ba25b4f8cb3b47d173e20ff3e: Bootstrap starting.
I20260620 19:12:55.163321 25699 tablet_bootstrap.cc:654] T 6ab3ae920c554934a7b36da4f8ee4a66 P 2dee97c110914202b2e9ca8db83df0bf: Neither blocks nor log segments found. Creating new log.
I20260620 19:12:55.163825 25700 tablet_bootstrap.cc:654] T 6ab3ae920c554934a7b36da4f8ee4a66 P 0b46980251d54a8bb398e0da234a85ec: Neither blocks nor log segments found. Creating new log.
I20260620 19:12:55.164424 25699 log.cc:824] T 6ab3ae920c554934a7b36da4f8ee4a66 P 2dee97c110914202b2e9ca8db83df0bf: Log is configured to *not* fsync() on all Append() calls
I20260620 19:12:55.164906 25701 tablet_bootstrap.cc:654] T 6ab3ae920c554934a7b36da4f8ee4a66 P a830c89ba25b4f8cb3b47d173e20ff3e: Neither blocks nor log segments found. Creating new log.
I20260620 19:12:55.165050 25700 log.cc:824] T 6ab3ae920c554934a7b36da4f8ee4a66 P 0b46980251d54a8bb398e0da234a85ec: Log is configured to *not* fsync() on all Append() calls
I20260620 19:12:55.165716 25701 log.cc:824] T 6ab3ae920c554934a7b36da4f8ee4a66 P a830c89ba25b4f8cb3b47d173e20ff3e: Log is configured to *not* fsync() on all Append() calls
I20260620 19:12:55.166797 25699 tablet_bootstrap.cc:492] T 6ab3ae920c554934a7b36da4f8ee4a66 P 2dee97c110914202b2e9ca8db83df0bf: No bootstrap required, opened a new log
I20260620 19:12:55.166986 25699 ts_tablet_manager.cc:1397] T 6ab3ae920c554934a7b36da4f8ee4a66 P 2dee97c110914202b2e9ca8db83df0bf: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20260620 19:12:55.167625 25701 tablet_bootstrap.cc:492] T 6ab3ae920c554934a7b36da4f8ee4a66 P a830c89ba25b4f8cb3b47d173e20ff3e: No bootstrap required, opened a new log
I20260620 19:12:55.167627 25700 tablet_bootstrap.cc:492] T 6ab3ae920c554934a7b36da4f8ee4a66 P 0b46980251d54a8bb398e0da234a85ec: No bootstrap required, opened a new log
I20260620 19:12:55.167804 25701 ts_tablet_manager.cc:1397] T 6ab3ae920c554934a7b36da4f8ee4a66 P a830c89ba25b4f8cb3b47d173e20ff3e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 19:12:55.167804 25700 ts_tablet_manager.cc:1397] T 6ab3ae920c554934a7b36da4f8ee4a66 P 0b46980251d54a8bb398e0da234a85ec: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 19:12:55.169948 25699 raft_consensus.cc:348] T 6ab3ae920c554934a7b36da4f8ee4a66 P 2dee97c110914202b2e9ca8db83df0bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b46980251d54a8bb398e0da234a85ec" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 38157 } } peers { permanent_uuid: "2dee97c110914202b2e9ca8db83df0bf" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 39169 } } peers { permanent_uuid: "a830c89ba25b4f8cb3b47d173e20ff3e" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 45875 } }
I20260620 19:12:55.170181 25699 raft_consensus.cc:374] T 6ab3ae920c554934a7b36da4f8ee4a66 P 2dee97c110914202b2e9ca8db83df0bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:12:55.170239 25699 raft_consensus.cc:729] T 6ab3ae920c554934a7b36da4f8ee4a66 P 2dee97c110914202b2e9ca8db83df0bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dee97c110914202b2e9ca8db83df0bf, State: Initialized, Role: FOLLOWER
I20260620 19:12:55.170652 25699 consensus_queue.cc:260] T 6ab3ae920c554934a7b36da4f8ee4a66 P 2dee97c110914202b2e9ca8db83df0bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b46980251d54a8bb398e0da234a85ec" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 38157 } } peers { permanent_uuid: "2dee97c110914202b2e9ca8db83df0bf" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 39169 } } peers { permanent_uuid: "a830c89ba25b4f8cb3b47d173e20ff3e" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 45875 } }
I20260620 19:12:55.171310 25414 heartbeater.cc:503] Master 127.9.155.126:40693 was elected leader, sending a full tablet report...
I20260620 19:12:55.171738 25699 ts_tablet_manager.cc:1428] T 6ab3ae920c554934a7b36da4f8ee4a66 P 2dee97c110914202b2e9ca8db83df0bf: Time spent starting tablet: real 0.005s user 0.003s sys 0.001s
I20260620 19:12:55.171592 25700 raft_consensus.cc:348] T 6ab3ae920c554934a7b36da4f8ee4a66 P 0b46980251d54a8bb398e0da234a85ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b46980251d54a8bb398e0da234a85ec" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 38157 } } peers { permanent_uuid: "2dee97c110914202b2e9ca8db83df0bf" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 39169 } } peers { permanent_uuid: "a830c89ba25b4f8cb3b47d173e20ff3e" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 45875 } }
I20260620 19:12:55.171592 25701 raft_consensus.cc:348] T 6ab3ae920c554934a7b36da4f8ee4a66 P a830c89ba25b4f8cb3b47d173e20ff3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b46980251d54a8bb398e0da234a85ec" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 38157 } } peers { permanent_uuid: "2dee97c110914202b2e9ca8db83df0bf" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 39169 } } peers { permanent_uuid: "a830c89ba25b4f8cb3b47d173e20ff3e" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 45875 } }
I20260620 19:12:55.171850 25700 raft_consensus.cc:374] T 6ab3ae920c554934a7b36da4f8ee4a66 P 0b46980251d54a8bb398e0da234a85ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:12:55.171850 25701 raft_consensus.cc:374] T 6ab3ae920c554934a7b36da4f8ee4a66 P a830c89ba25b4f8cb3b47d173e20ff3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:12:55.171898 25701 raft_consensus.cc:729] T 6ab3ae920c554934a7b36da4f8ee4a66 P a830c89ba25b4f8cb3b47d173e20ff3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a830c89ba25b4f8cb3b47d173e20ff3e, State: Initialized, Role: FOLLOWER
I20260620 19:12:55.171898 25700 raft_consensus.cc:729] T 6ab3ae920c554934a7b36da4f8ee4a66 P 0b46980251d54a8bb398e0da234a85ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b46980251d54a8bb398e0da234a85ec, State: Initialized, Role: FOLLOWER
I20260620 19:12:55.172492 25700 consensus_queue.cc:260] T 6ab3ae920c554934a7b36da4f8ee4a66 P 0b46980251d54a8bb398e0da234a85ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b46980251d54a8bb398e0da234a85ec" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 38157 } } peers { permanent_uuid: "2dee97c110914202b2e9ca8db83df0bf" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 39169 } } peers { permanent_uuid: "a830c89ba25b4f8cb3b47d173e20ff3e" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 45875 } }
I20260620 19:12:55.172514 25701 consensus_queue.cc:260] T 6ab3ae920c554934a7b36da4f8ee4a66 P a830c89ba25b4f8cb3b47d173e20ff3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b46980251d54a8bb398e0da234a85ec" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 38157 } } peers { permanent_uuid: "2dee97c110914202b2e9ca8db83df0bf" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 39169 } } peers { permanent_uuid: "a830c89ba25b4f8cb3b47d173e20ff3e" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 45875 } }
I20260620 19:12:55.173571 25701 ts_tablet_manager.cc:1428] T 6ab3ae920c554934a7b36da4f8ee4a66 P a830c89ba25b4f8cb3b47d173e20ff3e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 19:12:55.173928 25548 heartbeater.cc:503] Master 127.9.155.126:40693 was elected leader, sending a full tablet report...
I20260620 19:12:55.174036 25700 ts_tablet_manager.cc:1428] T 6ab3ae920c554934a7b36da4f8ee4a66 P 0b46980251d54a8bb398e0da234a85ec: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 19:12:55.174364 25682 heartbeater.cc:503] Master 127.9.155.126:40693 was elected leader, sending a full tablet report...
W20260620 19:12:55.185050 25415 tablet.cc:2369] T 6ab3ae920c554934a7b36da4f8ee4a66 P 2dee97c110914202b2e9ca8db83df0bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 19:12:55.351210 25683 tablet.cc:2369] T 6ab3ae920c554934a7b36da4f8ee4a66 P 0b46980251d54a8bb398e0da234a85ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 19:12:55.388067 25549 tablet.cc:2369] T 6ab3ae920c554934a7b36da4f8ee4a66 P a830c89ba25b4f8cb3b47d173e20ff3e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:12:55.584575 25707 raft_consensus.cc:482] T 6ab3ae920c554934a7b36da4f8ee4a66 P a830c89ba25b4f8cb3b47d173e20ff3e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:12:55.584810 25707 raft_consensus.cc:504] T 6ab3ae920c554934a7b36da4f8ee4a66 P a830c89ba25b4f8cb3b47d173e20ff3e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b46980251d54a8bb398e0da234a85ec" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 38157 } } peers { permanent_uuid: "2dee97c110914202b2e9ca8db83df0bf" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 39169 } } peers { permanent_uuid: "a830c89ba25b4f8cb3b47d173e20ff3e" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 45875 } }
I20260620 19:12:55.586004 25707 leader_election.cc:288] T 6ab3ae920c554934a7b36da4f8ee4a66 P a830c89ba25b4f8cb3b47d173e20ff3e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b46980251d54a8bb398e0da234a85ec (127.9.155.67:38157), 2dee97c110914202b2e9ca8db83df0bf (127.9.155.65:39169)
I20260620 19:12:55.589689 25695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:55.586349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.66:53063 (local address 127.9.155.67:38157)
0620 19:12:55.586481 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:55.586513 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:55.586619 (+ 106us) server_negotiation.cc:407] Connection header received
0620 19:12:55.586817 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:55.586821 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:55.586869 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:55.586930 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:12:55.587183 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:55.587689 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:55.588788 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:55.589177 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:55.589285 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:55.589337 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:55.589488 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 19:12:55.589556 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 19:12:55.589702 25708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:55.586257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.67:38157 (local address 127.9.155.66:53063)
0620 19:12:55.586527 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:55.586542 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:55.586701 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:55.586955 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:55.586958 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:55.586973 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:12:55.587083 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:55.587088 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:55.587795 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:55.587799 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:55.588672 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:55.588681 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:55.589288 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:55.589291 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:55.589381 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:55.589400 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 19:12:55.589462 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 19:12:55.589532 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":1,"thread_start_us":80,"threads_started":1}
I20260620 19:12:55.589725 25697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:55.586722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.66:43527 (local address 127.9.155.65:39169)
0620 19:12:55.586833 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:55.586837 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:55.586849 (+ 12us) server_negotiation.cc:407] Connection header received
0620 19:12:55.586897 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:55.586900 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:55.586940 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:55.587004 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:12:55.587277 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:55.587762 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:55.588824 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:55.589214 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:55.589317 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:55.589368 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:55.589543 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 19:12:55.589608 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 19:12:55.590065 25709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:55.586291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.65:39169 (local address 127.9.155.66:43527)
0620 19:12:55.586677 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:55.586690 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:55.586791 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:55.587041 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:55.587044 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:55.587061 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:12:55.587173 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:55.587179 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:55.587868 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:55.587871 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:55.588710 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:55.588719 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:55.589326 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:55.589329 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:55.589427 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:55.589444 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 19:12:55.589504 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 19:12:55.589571 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"mutex_wait_us":19,"thread_start_us":61,"threads_started":1}
I20260620 19:12:55.590416 25636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ab3ae920c554934a7b36da4f8ee4a66" candidate_uuid: "a830c89ba25b4f8cb3b47d173e20ff3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b46980251d54a8bb398e0da234a85ec" is_pre_election: true
I20260620 19:12:55.590699 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ab3ae920c554934a7b36da4f8ee4a66" candidate_uuid: "a830c89ba25b4f8cb3b47d173e20ff3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2dee97c110914202b2e9ca8db83df0bf" is_pre_election: true
I20260620 19:12:55.590782 25636 raft_consensus.cc:2454] T 6ab3ae920c554934a7b36da4f8ee4a66 P 0b46980251d54a8bb398e0da234a85ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a830c89ba25b4f8cb3b47d173e20ff3e in term 0.
I20260620 19:12:55.590968 25368 raft_consensus.cc:2454] T 6ab3ae920c554934a7b36da4f8ee4a66 P 2dee97c110914202b2e9ca8db83df0bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a830c89ba25b4f8cb3b47d173e20ff3e in term 0.
I20260620 19:12:55.591306 25436 leader_election.cc:302] T 6ab3ae920c554934a7b36da4f8ee4a66 P a830c89ba25b4f8cb3b47d173e20ff3e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b46980251d54a8bb398e0da234a85ec, a830c89ba25b4f8cb3b47d173e20ff3e; no voters:
I20260620 19:12:55.591585 25707 raft_consensus.cc:2781] T 6ab3ae920c554934a7b36da4f8ee4a66 P a830c89ba25b4f8cb3b47d173e20ff3e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:12:55.591676 25707 raft_consensus.cc:482] T 6ab3ae920c554934a7b36da4f8ee4a66 P a830c89ba25b4f8cb3b47d173e20ff3e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:12:55.591728 25707 raft_consensus.cc:3037] T 6ab3ae920c554934a7b36da4f8ee4a66 P a830c89ba25b4f8cb3b47d173e20ff3e [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:12:55.592888 25707 raft_consensus.cc:504] T 6ab3ae920c554934a7b36da4f8ee4a66 P a830c89ba25b4f8cb3b47d173e20ff3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b46980251d54a8bb398e0da234a85ec" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 38157 } } peers { permanent_uuid: "2dee97c110914202b2e9ca8db83df0bf" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 39169 } } peers { permanent_uuid: "a830c89ba25b4f8cb3b47d173e20ff3e" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 45875 } }
I20260620 19:12:55.593245 25707 leader_election.cc:288] T 6ab3ae920c554934a7b36da4f8ee4a66 P a830c89ba25b4f8cb3b47d173e20ff3e [CANDIDATE]: Term 1 election: Requested vote from peers 0b46980251d54a8bb398e0da234a85ec (127.9.155.67:38157), 2dee97c110914202b2e9ca8db83df0bf (127.9.155.65:39169)
I20260620 19:12:55.593544 25636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ab3ae920c554934a7b36da4f8ee4a66" candidate_uuid: "a830c89ba25b4f8cb3b47d173e20ff3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b46980251d54a8bb398e0da234a85ec"
I20260620 19:12:55.593657 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ab3ae920c554934a7b36da4f8ee4a66" candidate_uuid: "a830c89ba25b4f8cb3b47d173e20ff3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2dee97c110914202b2e9ca8db83df0bf"
I20260620 19:12:55.593662 25636 raft_consensus.cc:3037] T 6ab3ae920c554934a7b36da4f8ee4a66 P 0b46980251d54a8bb398e0da234a85ec [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:12:55.593784 25368 raft_consensus.cc:3037] T 6ab3ae920c554934a7b36da4f8ee4a66 P 2dee97c110914202b2e9ca8db83df0bf [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:12:55.594655 25636 raft_consensus.cc:2454] T 6ab3ae920c554934a7b36da4f8ee4a66 P 0b46980251d54a8bb398e0da234a85ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a830c89ba25b4f8cb3b47d173e20ff3e in term 1.
I20260620 19:12:55.594789 25368 raft_consensus.cc:2454] T 6ab3ae920c554934a7b36da4f8ee4a66 P 2dee97c110914202b2e9ca8db83df0bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a830c89ba25b4f8cb3b47d173e20ff3e in term 1.
I20260620 19:12:55.594946 25436 leader_election.cc:302] T 6ab3ae920c554934a7b36da4f8ee4a66 P a830c89ba25b4f8cb3b47d173e20ff3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b46980251d54a8bb398e0da234a85ec, a830c89ba25b4f8cb3b47d173e20ff3e; no voters:
I20260620 19:12:55.595172 25707 raft_consensus.cc:2781] T 6ab3ae920c554934a7b36da4f8ee4a66 P a830c89ba25b4f8cb3b47d173e20ff3e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:12:55.595345 25707 raft_consensus.cc:686] T 6ab3ae920c554934a7b36da4f8ee4a66 P a830c89ba25b4f8cb3b47d173e20ff3e [term 1 LEADER]: Becoming Leader. State: Replica: a830c89ba25b4f8cb3b47d173e20ff3e, State: Running, Role: LEADER
I20260620 19:12:55.595657 25707 consensus_queue.cc:237] T 6ab3ae920c554934a7b36da4f8ee4a66 P a830c89ba25b4f8cb3b47d173e20ff3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b46980251d54a8bb398e0da234a85ec" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 38157 } } peers { permanent_uuid: "2dee97c110914202b2e9ca8db83df0bf" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 39169 } } peers { permanent_uuid: "a830c89ba25b4f8cb3b47d173e20ff3e" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 45875 } }
I20260620 19:12:55.598635 25221 catalog_manager.cc:5495] T 6ab3ae920c554934a7b36da4f8ee4a66 P a830c89ba25b4f8cb3b47d173e20ff3e reported cstate change: term changed from 0 to 1, leader changed from <none> to a830c89ba25b4f8cb3b47d173e20ff3e (127.9.155.66). New cstate: current_term: 1 leader_uuid: "a830c89ba25b4f8cb3b47d173e20ff3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b46980251d54a8bb398e0da234a85ec" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 38157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2dee97c110914202b2e9ca8db83df0bf" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 39169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a830c89ba25b4f8cb3b47d173e20ff3e" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 45875 } health_report { overall_health: HEALTHY } } }
I20260620 19:12:55.711889 25714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:55.708091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47750 (local address 127.9.155.66:45875)
0620 19:12:55.708355 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:55.708360 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:55.708375 (+ 15us) server_negotiation.cc:407] Connection header received
0620 19:12:55.708497 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:55.708499 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:55.708566 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:55.708679 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:12:55.708929 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:55.709461 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:55.710342 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:55.710581 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:55.710669 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:55.711012 (+ 343us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:12:55.711473 (+ 461us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:12:55.711595 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:55.711671 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 19:12:55.711730 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":63,"threads_started":1}
I20260620 19:12:55.717767 25636 raft_consensus.cc:1261] T 6ab3ae920c554934a7b36da4f8ee4a66 P 0b46980251d54a8bb398e0da234a85ec [term 1 FOLLOWER]: Refusing update from remote peer a830c89ba25b4f8cb3b47d173e20ff3e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:12:55.717767 25368 raft_consensus.cc:1261] T 6ab3ae920c554934a7b36da4f8ee4a66 P 2dee97c110914202b2e9ca8db83df0bf [term 1 FOLLOWER]: Refusing update from remote peer a830c89ba25b4f8cb3b47d173e20ff3e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:12:55.718638 25710 consensus_queue.cc:1035] T 6ab3ae920c554934a7b36da4f8ee4a66 P a830c89ba25b4f8cb3b47d173e20ff3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b46980251d54a8bb398e0da234a85ec" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 38157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:12:55.718842 25707 consensus_queue.cc:1035] T 6ab3ae920c554934a7b36da4f8ee4a66 P a830c89ba25b4f8cb3b47d173e20ff3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dee97c110914202b2e9ca8db83df0bf" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 39169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:12:55.736814 25221 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52196:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:12:55.736982 25221 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52196:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:12:55.739341 25221 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 025239f00c2441fe885affb9769996d9: Sending DeleteTablet for 3 replicas of tablet 6ab3ae920c554934a7b36da4f8ee4a66
I20260620 19:12:55.740084 25616 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ab3ae920c554934a7b36da4f8ee4a66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:12:55 UTC) from {username='kudu', principal='kudu/127.9.155.126@KRBTEST.COM'} at 127.0.0.1:54682
I20260620 19:12:55.740358 25348 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ab3ae920c554934a7b36da4f8ee4a66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:12:55 UTC) from {username='kudu', principal='kudu/127.9.155.126@KRBTEST.COM'} at 127.0.0.1:59102
I20260620 19:12:55.740463 25482 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ab3ae920c554934a7b36da4f8ee4a66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:12:55 UTC) from {username='kudu', principal='kudu/127.9.155.126@KRBTEST.COM'} at 127.0.0.1:47738
I20260620 19:12:55.740957 25729 tablet_replica.cc:330] T 6ab3ae920c554934a7b36da4f8ee4a66 P a830c89ba25b4f8cb3b47d173e20ff3e: stopping tablet replica
I20260620 19:12:55.741092 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 25285
I20260620 19:12:55.741285 25729 raft_consensus.cc:2229] T 6ab3ae920c554934a7b36da4f8ee4a66 P a830c89ba25b4f8cb3b47d173e20ff3e [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:12:55.741781 25729 raft_consensus.cc:2258] T 6ab3ae920c554934a7b36da4f8ee4a66 P a830c89ba25b4f8cb3b47d173e20ff3e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:12:55.743052 25729 ts_tablet_manager.cc:1905] T 6ab3ae920c554934a7b36da4f8ee4a66 P a830c89ba25b4f8cb3b47d173e20ff3e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:12:55.744168 25727 tablet_replica.cc:330] T 6ab3ae920c554934a7b36da4f8ee4a66 P 0b46980251d54a8bb398e0da234a85ec: stopping tablet replica
I20260620 19:12:55.744515 25727 raft_consensus.cc:2229] T 6ab3ae920c554934a7b36da4f8ee4a66 P 0b46980251d54a8bb398e0da234a85ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:12:55.744851 25727 raft_consensus.cc:2258] T 6ab3ae920c554934a7b36da4f8ee4a66 P 0b46980251d54a8bb398e0da234a85ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:12:55.745817 25729 ts_tablet_manager.cc:1918] T 6ab3ae920c554934a7b36da4f8ee4a66 P a830c89ba25b4f8cb3b47d173e20ff3e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:12:55.745915 25729 log.cc:1196] T 6ab3ae920c554934a7b36da4f8ee4a66 P a830c89ba25b4f8cb3b47d173e20ff3e: Deleting WAL directory at /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-1/wal/wals/6ab3ae920c554934a7b36da4f8ee4a66
I20260620 19:12:55.746212 25729 ts_tablet_manager.cc:1939] T 6ab3ae920c554934a7b36da4f8ee4a66 P a830c89ba25b4f8cb3b47d173e20ff3e: Deleting consensus metadata
I20260620 19:12:55.746284 25727 ts_tablet_manager.cc:1905] T 6ab3ae920c554934a7b36da4f8ee4a66 P 0b46980251d54a8bb398e0da234a85ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:12:55.747445 25209 catalog_manager.cc:4841] TS a830c89ba25b4f8cb3b47d173e20ff3e (127.9.155.66:45875): tablet 6ab3ae920c554934a7b36da4f8ee4a66 (table test-table [id=2a607240899b4b9e993b7a383a4f52ec]) successfully deleted
I20260620 19:12:55.748688 25727 ts_tablet_manager.cc:1918] T 6ab3ae920c554934a7b36da4f8ee4a66 P 0b46980251d54a8bb398e0da234a85ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:12:55.748775 25727 log.cc:1196] T 6ab3ae920c554934a7b36da4f8ee4a66 P 0b46980251d54a8bb398e0da234a85ec: Deleting WAL directory at /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-2/wal/wals/6ab3ae920c554934a7b36da4f8ee4a66
I20260620 19:12:55.749028 25727 ts_tablet_manager.cc:1939] T 6ab3ae920c554934a7b36da4f8ee4a66 P 0b46980251d54a8bb398e0da234a85ec: Deleting consensus metadata
W20260620 19:12:55.749766 25207 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.155.65:39169 (error 108)
I20260620 19:12:55.749992 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 25419
I20260620 19:12:55.750485 25208 catalog_manager.cc:4841] TS 0b46980251d54a8bb398e0da234a85ec (127.9.155.67:38157): tablet 6ab3ae920c554934a7b36da4f8ee4a66 (table test-table [id=2a607240899b4b9e993b7a383a4f52ec]) successfully deleted
I20260620 19:12:55.751562 25731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:55.751052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.65:39169 (local address 127.0.0.1:59106)
0620 19:12:55.751390 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:55.751461 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.65:39169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":50,"threads_started":1}
W20260620 19:12:55.751732 25207 catalog_manager.cc:4584] TS 2dee97c110914202b2e9ca8db83df0bf (127.9.155.65:39169): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6ab3ae920c554934a7b36da4f8ee4a66: Network error: Client connection negotiation failed: client connection to 127.9.155.65:39169: connect: Connection refused (error 111)
I20260620 19:12:55.755609 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 25553
I20260620 19:12:55.760335 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 25191
I20260620 19:13:15.766089 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:40693
--webserver_interface=127.9.155.126
--webserver_port=40719
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab
--principal=kudu/127.9.155.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:42519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.155.126:40693
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:13:15.892954 25732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:13:15.893288 25732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:13:15.893348 25732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:13:15.897526 25732 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 19:13:15.897606 25732 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 19:13:15.897629 25732 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 19:13:15.897647 25732 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:13:15.897670 25732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:13:15.897694 25732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:13:15.897732 25732 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:13:15.897758 25732 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:13:15.903664 25732 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:42519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.9.155.126:40693
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab
--principal=kudu/127.9.155.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:40693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.155.126
--webserver_port=40719
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25732
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:13:15.905155 25732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:13:15.906479 25732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:13:15.921581 25739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:13:15.921622 25738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:13:15.921729 25741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:13:15.921802 25732 server_base.cc:1034] running on GCE node
I20260620 19:13:15.922683 25732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:13:15.924098 25732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:13:15.925283 25732 hybrid_clock.cc:648] HybridClock initialized: now 1781982795925262 us; error 39 us; skew 500 ppm
Jun 20 19:13:15 dist-test-slave-n3fx krb5kdc[25178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982795, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:13:15.928495 25732 init.cc:377] Logged in from keytab as kudu/127.9.155.126@KRBTEST.COM (short username kudu)
I20260620 19:13:15.929543 25732 webserver.cc:460] Webserver started at http://127.9.155.126:40719/ using document root <none> and password file <none>
I20260620 19:13:15.930191 25732 fs_manager.cc:359] Metadata directory not provided
I20260620 19:13:15.930289 25732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:13:15.934885 25732 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 19:13:15.937364 25748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:15.938849 25732 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:13:15.938988 25732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/master-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/master-0/wal
uuid: "025239f00c2441fe885affb9769996d9"
format_stamp: "Formatted at 2026-06-20 19:12:54 on dist-test-slave-n3fx"
server_key: "0c722aadc3be79eeb06f6000b03a0746"
server_key_iv: "a9c7db7ad71d08c9f106596a6f7c0434"
server_key_version: "encryptionkey@0"
I20260620 19:13:15.939705 25732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:13:15.952517 25732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:13:15.956820 25732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:13:15.957050 25732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:13:15.964288 25732 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.126:40693
I20260620 19:13:15.964594 25800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.126:40693 every 8 connection(s)
I20260620 19:13:15.965629 25732 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
I20260620 19:13:15.970718 25801 sys_catalog.cc:263] Verifying existing consensus state
I20260620 19:13:15.972738 25801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 025239f00c2441fe885affb9769996d9: Bootstrap starting.
I20260620 19:13:15.974385 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 25732
I20260620 19:13:15.974901 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.65:39169
--local_ip_for_outbound_sockets=127.9.155.65
--tserver_master_addrs=127.9.155.126:40693
--webserver_port=36173
--webserver_interface=127.9.155.65
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.65.keytab
--principal=kudu/127.9.155.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:42519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:13:15.990931 25801 log.cc:824] T 00000000000000000000000000000000 P 025239f00c2441fe885affb9769996d9: Log is configured to *not* fsync() on all Append() calls
I20260620 19:13:16.000305 25801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 025239f00c2441fe885affb9769996d9: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:13:16.000671 25801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 025239f00c2441fe885affb9769996d9: Bootstrap complete.
I20260620 19:13:16.004704 25801 raft_consensus.cc:348] T 00000000000000000000000000000000 P 025239f00c2441fe885affb9769996d9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "025239f00c2441fe885affb9769996d9" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 40693 } }
I20260620 19:13:16.005148 25801 raft_consensus.cc:729] T 00000000000000000000000000000000 P 025239f00c2441fe885affb9769996d9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 025239f00c2441fe885affb9769996d9, State: Initialized, Role: FOLLOWER
I20260620 19:13:16.005674 25801 consensus_queue.cc:260] T 00000000000000000000000000000000 P 025239f00c2441fe885affb9769996d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "025239f00c2441fe885affb9769996d9" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 40693 } }
I20260620 19:13:16.005800 25801 raft_consensus.cc:388] T 00000000000000000000000000000000 P 025239f00c2441fe885affb9769996d9 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:13:16.005874 25801 raft_consensus.cc:482] T 00000000000000000000000000000000 P 025239f00c2441fe885affb9769996d9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:13:16.005967 25801 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 025239f00c2441fe885affb9769996d9 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:13:16.008026 25801 raft_consensus.cc:504] T 00000000000000000000000000000000 P 025239f00c2441fe885affb9769996d9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "025239f00c2441fe885affb9769996d9" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 40693 } }
I20260620 19:13:16.008443 25801 leader_election.cc:302] T 00000000000000000000000000000000 P 025239f00c2441fe885affb9769996d9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 025239f00c2441fe885affb9769996d9; no voters:
I20260620 19:13:16.008767 25801 leader_election.cc:288] T 00000000000000000000000000000000 P 025239f00c2441fe885affb9769996d9 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 19:13:16.008872 25805 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 025239f00c2441fe885affb9769996d9 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:13:16.009153 25805 raft_consensus.cc:686] T 00000000000000000000000000000000 P 025239f00c2441fe885affb9769996d9 [term 2 LEADER]: Becoming Leader. State: Replica: 025239f00c2441fe885affb9769996d9, State: Running, Role: LEADER
I20260620 19:13:16.009542 25805 consensus_queue.cc:237] T 00000000000000000000000000000000 P 025239f00c2441fe885affb9769996d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "025239f00c2441fe885affb9769996d9" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 40693 } }
I20260620 19:13:16.010083 25801 sys_catalog.cc:564] T 00000000000000000000000000000000 P 025239f00c2441fe885affb9769996d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:13:16.010951 25806 sys_catalog.cc:455] T 00000000000000000000000000000000 P 025239f00c2441fe885affb9769996d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "025239f00c2441fe885affb9769996d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "025239f00c2441fe885affb9769996d9" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 40693 } } }
I20260620 19:13:16.011080 25806 sys_catalog.cc:458] T 00000000000000000000000000000000 P 025239f00c2441fe885affb9769996d9 [sys.catalog]: This master's current role is: LEADER
I20260620 19:13:16.011361 25807 sys_catalog.cc:455] T 00000000000000000000000000000000 P 025239f00c2441fe885affb9769996d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 025239f00c2441fe885affb9769996d9. Latest consensus state: current_term: 2 leader_uuid: "025239f00c2441fe885affb9769996d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "025239f00c2441fe885affb9769996d9" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 40693 } } }
I20260620 19:13:16.011442 25807 sys_catalog.cc:458] T 00000000000000000000000000000000 P 025239f00c2441fe885affb9769996d9 [sys.catalog]: This master's current role is: LEADER
I20260620 19:13:16.015431 25821 catalog_manager.cc:1259] Loaded cluster ID: 36f418d264f64c6fb041fa4533bace46
I20260620 19:13:16.015501 25821 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 025239f00c2441fe885affb9769996d9: loading cluster ID for follower catalog manager: success
I20260620 19:13:16.018024 25821 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 025239f00c2441fe885affb9769996d9: acquiring CA information for follower catalog manager: success
I20260620 19:13:16.019523 25821 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 025239f00c2441fe885affb9769996d9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 19:13:16.019793 25814 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:13:16.022132 25814 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:13:16.022617 25814 catalog_manager.cc:1259] Loaded cluster ID: 36f418d264f64c6fb041fa4533bace46
I20260620 19:13:16.022672 25814 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:13:16.023403 25814 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:13:16.023918 25814 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 025239f00c2441fe885affb9769996d9: Loaded TSK: 0
I20260620 19:13:16.042850 25814 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 025239f00c2441fe885affb9769996d9: Generated new TSK 1
I20260620 19:13:16.043175 25814 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 19:13:16.104800 25803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:13:16.105099 25803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:13:16.105165 25803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:13:16.109424 25803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:13:16.109516 25803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:13:16.109601 25803 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.65
I20260620 19:13:16.115602 25803 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:42519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.65.keytab
--principal=kudu/127.9.155.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.65:39169
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.155.65
--webserver_port=36173
--tserver_master_addrs=127.9.155.126:40693
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25803
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.65
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:13:16.116876 25803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:13:16.117952 25803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:13:16.128031 25828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:13:16.128130 25831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:13:16.128044 25829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:13:16.129120 25803 server_base.cc:1034] running on GCE node
I20260620 19:13:16.129631 25803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:13:16.130343 25803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:13:16.131575 25803 hybrid_clock.cc:648] HybridClock initialized: now 1781982796131486 us; error 52 us; skew 500 ppm
Jun 20 19:13:16 dist-test-slave-n3fx krb5kdc[25178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982796, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:13:16.134660 25803 init.cc:377] Logged in from keytab as kudu/127.9.155.65@KRBTEST.COM (short username kudu)
I20260620 19:13:16.135674 25803 webserver.cc:460] Webserver started at http://127.9.155.65:36173/ using document root <none> and password file <none>
I20260620 19:13:16.136309 25803 fs_manager.cc:359] Metadata directory not provided
I20260620 19:13:16.136399 25803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:13:16.140595 25803 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 19:13:16.142982 25838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:16.144515 25803 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:13:16.144639 25803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-0/wal
uuid: "2dee97c110914202b2e9ca8db83df0bf"
format_stamp: "Formatted at 2026-06-20 19:12:54 on dist-test-slave-n3fx"
server_key: "6c811a05433d6736303f0a3060afa774"
server_key_iv: "d75bd652d3fdee25dbbd0add8a9744a9"
server_key_version: "encryptionkey@0"
I20260620 19:13:16.145208 25803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:13:16.180282 25803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:13:16.183537 25803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:13:16.183714 25803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:13:16.184360 25803 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:13:16.185513 25845 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 19:13:16.189661 25803 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 19:13:16.189738 25803 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260620 19:13:16.189802 25803 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 19:13:16.191567 25803 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 19:13:16.191618 25803 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260620 19:13:16.191921 25845 tablet_bootstrap.cc:492] T 6ab3ae920c554934a7b36da4f8ee4a66 P 2dee97c110914202b2e9ca8db83df0bf: Bootstrap starting.
I20260620 19:13:16.201211 25803 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.65:39169
I20260620 19:13:16.201285 25952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.65:39169 every 8 connection(s)
I20260620 19:13:16.202188 25803 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
I20260620 19:13:16.209875 25845 log.cc:824] T 6ab3ae920c554934a7b36da4f8ee4a66 P 2dee97c110914202b2e9ca8db83df0bf: Log is configured to *not* fsync() on all Append() calls
I20260620 19:13:16.212460 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 25803
I20260620 19:13:16.212867 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.66:45875
--local_ip_for_outbound_sockets=127.9.155.66
--tserver_master_addrs=127.9.155.126:40693
--webserver_port=45103
--webserver_interface=127.9.155.66
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.66.keytab
--principal=kudu/127.9.155.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:42519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 19:13:16 dist-test-slave-n3fx krb5kdc[25178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982796, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.65@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:13:16.218291 25845 tablet_bootstrap.cc:492] T 6ab3ae920c554934a7b36da4f8ee4a66 P 2dee97c110914202b2e9ca8db83df0bf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:13:16.218719 25845 tablet_bootstrap.cc:492] T 6ab3ae920c554934a7b36da4f8ee4a66 P 2dee97c110914202b2e9ca8db83df0bf: Bootstrap complete.
I20260620 19:13:16.219870 25845 ts_tablet_manager.cc:1397] T 6ab3ae920c554934a7b36da4f8ee4a66 P 2dee97c110914202b2e9ca8db83df0bf: Time spent bootstrapping tablet: real 0.028s user 0.016s sys 0.008s
I20260620 19:13:16.221580 25956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:16.204069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40693 (local address 127.9.155.65:50455)
0620 19:13:16.204668 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:16.204734 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:16.205863 (+ 1129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:16.207090 (+ 1227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:16.207099 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:16.207644 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:16.208362 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:16.208384 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:16.209977 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:16.209982 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:16.211183 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:16.211194 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:16.212104 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:16.212108 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:16.212319 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:16.213163 (+ 844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:16.213185 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:16.214978 (+ 1793us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:13:16.217769 (+ 2791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:16.217775 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:16.217788 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:16.218137 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:16.218600 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:16.218604 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:16.218605 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:16.218776 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:16.219502 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:13:16.219510 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:13:16.219845 (+ 335us) client_negotiation.cc:769] Sending connection context
0620 19:13:16.220076 (+ 231us) client_negotiation.cc:240] Negotiation successful
0620 19:13:16.220506 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":137,"threads_started":1}
I20260620 19:13:16.221593 25955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:16.204089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.65:50455 (local address 127.9.155.126:40693)
0620 19:13:16.204458 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:16.204478 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:16.204866 (+ 388us) server_negotiation.cc:407] Connection header received
0620 19:13:16.206304 (+ 1438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:16.206315 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:16.206686 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:16.206955 (+ 269us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:16.208618 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:16.209808 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:16.211322 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:16.211947 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:16.212078 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:16.215217 (+ 3139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:13:16.215223 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:13:16.215242 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:13:16.215280 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:13:16.217617 (+ 2337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:16.218294 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:16.218301 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:16.218308 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:16.218491 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:16.218897 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:16.218901 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:16.218903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:16.219370 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:13:16.219513 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:16.220355 (+ 842us) server_negotiation.cc:299] Negotiation successful
0620 19:13:16.220688 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":129,"threads_started":1}
I20260620 19:13:16.222899 25845 raft_consensus.cc:348] T 6ab3ae920c554934a7b36da4f8ee4a66 P 2dee97c110914202b2e9ca8db83df0bf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b46980251d54a8bb398e0da234a85ec" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 38157 } } peers { permanent_uuid: "2dee97c110914202b2e9ca8db83df0bf" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 39169 } } peers { permanent_uuid: "a830c89ba25b4f8cb3b47d173e20ff3e" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 45875 } }
I20260620 19:13:16.223321 25845 raft_consensus.cc:729] T 6ab3ae920c554934a7b36da4f8ee4a66 P 2dee97c110914202b2e9ca8db83df0bf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dee97c110914202b2e9ca8db83df0bf, State: Initialized, Role: FOLLOWER
I20260620 19:13:16.224613 25953 heartbeater.cc:347] Connected to a master server at 127.9.155.126:40693
I20260620 19:13:16.224608 25845 consensus_queue.cc:260] T 6ab3ae920c554934a7b36da4f8ee4a66 P 2dee97c110914202b2e9ca8db83df0bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b46980251d54a8bb398e0da234a85ec" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 38157 } } peers { permanent_uuid: "2dee97c110914202b2e9ca8db83df0bf" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 39169 } } peers { permanent_uuid: "a830c89ba25b4f8cb3b47d173e20ff3e" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 45875 } }
I20260620 19:13:16.224911 25953 heartbeater.cc:464] Registering TS with master...
I20260620 19:13:16.225492 25953 heartbeater.cc:511] Master 127.9.155.126:40693 requested a full tablet report, sending...
I20260620 19:13:16.225611 25845 ts_tablet_manager.cc:1428] T 6ab3ae920c554934a7b36da4f8ee4a66 P 2dee97c110914202b2e9ca8db83df0bf: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260620 19:13:16.228238 25765 ts_manager.cc:194] Registered new tserver with Master: 2dee97c110914202b2e9ca8db83df0bf (127.9.155.65:39169)
I20260620 19:13:16.231874 25765 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.155.65@KRBTEST.COM'} at 127.9.155.65:50455
I20260620 19:13:16.234767 25953 heartbeater.cc:503] Master 127.9.155.126:40693 was elected leader, sending a full tablet report...
Jun 20 19:13:16 dist-test-slave-n3fx krb5kdc[25178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982795, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.126@KRBTEST.COM for kudu/127.9.155.65@KRBTEST.COM
I20260620 19:13:16.242074 25960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:16.231239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.65:39169 (local address 127.0.0.1:48132)
0620 19:13:16.232127 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:16.232179 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:16.232350 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:16.232922 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:16.232928 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:16.233304 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:16.233412 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:16.233419 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:16.234361 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:16.234364 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:16.235028 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:16.235039 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:16.235664 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:16.235668 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:16.235782 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:16.236316 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:16.236332 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:16.238114 (+ 1782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:13:16.240033 (+ 1919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:16.240045 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:16.240051 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:16.240413 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:16.240827 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:16.240830 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:16.240832 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:16.240934 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:16.241385 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:13:16.241391 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:13:16.241529 (+ 138us) client_negotiation.cc:769] Sending connection context
0620 19:13:16.241666 (+ 137us) client_negotiation.cc:240] Negotiation successful
0620 19:13:16.241889 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"thread_start_us":117,"threads_started":1}
I20260620 19:13:16.242125 25961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:16.231361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48132 (local address 127.9.155.65:39169)
0620 19:13:16.231766 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:16.231778 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:16.232322 (+ 544us) server_negotiation.cc:407] Connection header received
0620 19:13:16.232504 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:16.232510 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:16.232768 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:16.232915 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:16.233533 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:16.234249 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:16.235137 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:16.235556 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:16.235661 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:16.238265 (+ 2604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:13:16.238273 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:13:16.238284 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:13:16.238309 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:13:16.239916 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:16.240563 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:16.240570 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:16.240573 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:16.240735 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:16.241021 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:16.241023 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:16.241024 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:16.241285 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:13:16.241385 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:16.241831 (+ 446us) server_negotiation.cc:299] Negotiation successful
0620 19:13:16.241983 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":104,"threads_started":1}
I20260620 19:13:16.243186 25887 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ab3ae920c554934a7b36da4f8ee4a66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:12:55 UTC) from {username='kudu', principal='kudu/127.9.155.126@KRBTEST.COM'} at 127.0.0.1:48132
I20260620 19:13:16.244309 25962 tablet_replica.cc:330] stopping tablet replica
I20260620 19:13:16.244701 25962 raft_consensus.cc:2229] T 6ab3ae920c554934a7b36da4f8ee4a66 P 2dee97c110914202b2e9ca8db83df0bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:13:16.244951 25962 raft_consensus.cc:2258] T 6ab3ae920c554934a7b36da4f8ee4a66 P 2dee97c110914202b2e9ca8db83df0bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:13:16.245893 25962 ts_tablet_manager.cc:1905] T 6ab3ae920c554934a7b36da4f8ee4a66 P 2dee97c110914202b2e9ca8db83df0bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:13:16.249626 25962 ts_tablet_manager.cc:1918] T 6ab3ae920c554934a7b36da4f8ee4a66 P 2dee97c110914202b2e9ca8db83df0bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:13:16.249715 25962 log.cc:1196] T 6ab3ae920c554934a7b36da4f8ee4a66 P 2dee97c110914202b2e9ca8db83df0bf: Deleting WAL directory at /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-0/wal/wals/6ab3ae920c554934a7b36da4f8ee4a66
I20260620 19:13:16.249927 25962 ts_tablet_manager.cc:1939] T 6ab3ae920c554934a7b36da4f8ee4a66 P 2dee97c110914202b2e9ca8db83df0bf: Deleting consensus metadata
I20260620 19:13:16.251840 25749 catalog_manager.cc:4841] TS 2dee97c110914202b2e9ca8db83df0bf (127.9.155.65:39169): tablet 6ab3ae920c554934a7b36da4f8ee4a66 (table test-table [id=2a607240899b4b9e993b7a383a4f52ec]) successfully deleted
W20260620 19:13:16.332420 25958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:13:16.332682 25958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:13:16.332726 25958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:13:16.337170 25958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:13:16.337244 25958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:13:16.337318 25958 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.66
I20260620 19:13:16.342948 25958 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:42519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.66.keytab
--principal=kudu/127.9.155.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.66:45875
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.155.66
--webserver_port=45103
--tserver_master_addrs=127.9.155.126:40693
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.66
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:13:16.344079 25958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:13:16.345142 25958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:13:16.354741 25970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:13:16.354818 25958 server_base.cc:1034] running on GCE node
W20260620 19:13:16.354830 25967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:13:16.354745 25968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:13:16.355376 25958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:13:16.356061 25958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:13:16.357226 25958 hybrid_clock.cc:648] HybridClock initialized: now 1781982796357200 us; error 40 us; skew 500 ppm
Jun 20 19:13:16 dist-test-slave-n3fx krb5kdc[25178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982796, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:13:16.359956 25958 init.cc:377] Logged in from keytab as kudu/127.9.155.66@KRBTEST.COM (short username kudu)
I20260620 19:13:16.360849 25958 webserver.cc:460] Webserver started at http://127.9.155.66:45103/ using document root <none> and password file <none>
I20260620 19:13:16.361377 25958 fs_manager.cc:359] Metadata directory not provided
I20260620 19:13:16.361455 25958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:13:16.365244 25958 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260620 19:13:16.367254 25977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:16.368515 25958 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:13:16.368626 25958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-1/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-1/wal
uuid: "a830c89ba25b4f8cb3b47d173e20ff3e"
format_stamp: "Formatted at 2026-06-20 19:12:54 on dist-test-slave-n3fx"
server_key: "5443a0cb3a88e72d78fd8b772e429b51"
server_key_iv: "6d83aac77ad59a26c37f02f9f4b7f1bc"
server_key_version: "encryptionkey@0"
I20260620 19:13:16.369197 25958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:13:16.384346 25958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:13:16.387459 25958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:13:16.387636 25958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:13:16.388239 25958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:13:16.389081 25958 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:13:16.389142 25958 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:16.389190 25958 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:13:16.389212 25958 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:16.397583 25958 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.66:45875
I20260620 19:13:16.397667 26090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.66:45875 every 8 connection(s)
I20260620 19:13:16.398595 25958 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
I20260620 19:13:16.399437 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 25958
I20260620 19:13:16.399816 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.67:38157
--local_ip_for_outbound_sockets=127.9.155.67
--tserver_master_addrs=127.9.155.126:40693
--webserver_port=45161
--webserver_interface=127.9.155.67
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.67.keytab
--principal=kudu/127.9.155.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:42519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 19:13:16 dist-test-slave-n3fx krb5kdc[25178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982796, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.66@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:13:16.413739 25955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:16.400473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.66:43743 (local address 127.9.155.126:40693)
0620 19:13:16.400632 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:16.400636 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:16.401582 (+ 946us) server_negotiation.cc:407] Connection header received
0620 19:13:16.402896 (+ 1314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:16.402899 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:16.402956 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:16.403064 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:16.404367 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:16.405208 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:16.406475 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:16.406845 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:16.406963 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:16.409801 (+ 2838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:13:16.409805 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:13:16.409808 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:13:16.409840 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:13:16.411529 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:16.412092 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:16.412095 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:16.412097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:16.412185 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:16.412555 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:16.412558 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:16.412560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:16.412785 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:13:16.412876 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:16.413403 (+ 527us) server_negotiation.cc:299] Negotiation successful
0620 19:13:16.413571 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 19:13:16.414584 26094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:16.400920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40693 (local address 127.9.155.66:43743)
0620 19:13:16.401406 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:16.401458 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:16.402671 (+ 1213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:16.403262 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:16.403269 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:16.403686 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:16.404178 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:16.404193 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:16.405358 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:16.405361 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:16.406334 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:16.406344 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:16.407003 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:16.407006 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:16.407169 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:16.407868 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:16.407886 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:16.409618 (+ 1732us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:13:16.411646 (+ 2028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:16.411653 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:16.411671 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:16.412003 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:16.412283 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:16.412285 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:16.412287 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:16.412433 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:16.412904 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:13:16.412910 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:13:16.413184 (+ 274us) client_negotiation.cc:769] Sending connection context
0620 19:13:16.413355 (+ 171us) client_negotiation.cc:240] Negotiation successful
0620 19:13:16.413666 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":130,"threads_started":1}
I20260620 19:13:16.415692 26091 heartbeater.cc:347] Connected to a master server at 127.9.155.126:40693
I20260620 19:13:16.415912 26091 heartbeater.cc:464] Registering TS with master...
I20260620 19:13:16.416447 26091 heartbeater.cc:511] Master 127.9.155.126:40693 requested a full tablet report, sending...
I20260620 19:13:16.417455 25765 ts_manager.cc:194] Registered new tserver with Master: a830c89ba25b4f8cb3b47d173e20ff3e (127.9.155.66:45875)
I20260620 19:13:16.418047 25765 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.155.66@KRBTEST.COM'} at 127.9.155.66:43743
W20260620 19:13:16.511552 26093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:13:16.511855 26093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:13:16.511920 26093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:13:16.516161 26093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:13:16.516264 26093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:13:16.516353 26093 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.67
I20260620 19:13:16.521984 26093 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:42519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.67.keytab
--principal=kudu/127.9.155.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.67:38157
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.155.67
--webserver_port=45161
--tserver_master_addrs=127.9.155.126:40693
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26093
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.67
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:13:16.523080 26093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:13:16.524096 26093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:13:16.533942 26099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:13:16.533941 26102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:13:16.533984 26093 server_base.cc:1034] running on GCE node
W20260620 19:13:16.533942 26100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:13:16.534651 26093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:13:16.535336 26093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:13:16.536514 26093 hybrid_clock.cc:648] HybridClock initialized: now 1781982796536492 us; error 32 us; skew 500 ppm
Jun 20 19:13:16 dist-test-slave-n3fx krb5kdc[25178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982796, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:13:16.539489 26093 init.cc:377] Logged in from keytab as kudu/127.9.155.67@KRBTEST.COM (short username kudu)
I20260620 19:13:16.540498 26093 webserver.cc:460] Webserver started at http://127.9.155.67:45161/ using document root <none> and password file <none>
I20260620 19:13:16.541044 26093 fs_manager.cc:359] Metadata directory not provided
I20260620 19:13:16.541138 26093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:13:16.545372 26093 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 19:13:16.548117 26109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:16.549650 26093 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:13:16.549788 26093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-2/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-2/wal
uuid: "0b46980251d54a8bb398e0da234a85ec"
format_stamp: "Formatted at 2026-06-20 19:12:55 on dist-test-slave-n3fx"
server_key: "d1c948a36f6cce92e06df2322361459e"
server_key_iv: "4557675b576f09102bea8534fc65fb9e"
server_key_version: "encryptionkey@0"
I20260620 19:13:16.550484 26093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:13:16.566538 26093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:13:16.569752 26093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:13:16.569932 26093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:13:16.570531 26093 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:13:16.571363 26093 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:13:16.571410 26093 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:16.571462 26093 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:13:16.571488 26093 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:16.579879 26093 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.67:38157
I20260620 19:13:16.579921 26222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.67:38157 every 8 connection(s)
I20260620 19:13:16.580926 26093 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
I20260620 19:13:16.586491 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 26093
Jun 20 19:13:16 dist-test-slave-n3fx krb5kdc[25178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982796, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.67@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:13:16.595230 26226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:16.587093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51358 (local address 127.9.155.126:40693)
0620 19:13:16.587321 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:16.587325 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:16.587475 (+ 150us) server_negotiation.cc:407] Connection header received
0620 19:13:16.587583 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:16.587586 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:16.587633 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:16.587713 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:16.588388 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:16.589009 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:16.589866 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:16.590223 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:16.590312 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:16.591862 (+ 1550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:13:16.591865 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:13:16.591868 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:13:16.591905 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:13:16.593547 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:16.593982 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:16.593986 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:16.593988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:16.594105 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:16.594352 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:16.594354 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:16.594356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:16.594527 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:13:16.594721 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:16.594872 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 19:13:16.595031 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":82,"threads_started":1}
I20260620 19:13:16.595451 25955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:16.582915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.67:41233 (local address 127.9.155.126:40693)
0620 19:13:16.583043 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:16.583047 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:16.583935 (+ 888us) server_negotiation.cc:407] Connection header received
0620 19:13:16.585212 (+ 1277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:16.585215 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:16.585272 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:16.585347 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:16.586482 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:16.587105 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:16.588434 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:16.588684 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:16.588764 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:16.591508 (+ 2744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:13:16.591511 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:13:16.591513 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:13:16.591544 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:13:16.593276 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:16.593798 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:16.593801 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:16.593803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:16.593886 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:16.594299 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:16.594302 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:16.594304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:16.594497 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:13:16.594622 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:16.595206 (+ 584us) server_negotiation.cc:299] Negotiation successful
0620 19:13:16.595304 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 19:13:16.596292 26225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:16.583246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40693 (local address 127.9.155.67:41233)
0620 19:13:16.583741 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:16.583782 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:16.585011 (+ 1229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:16.585533 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:16.585540 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:16.585876 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:16.586332 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:16.586345 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:16.587225 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:16.587229 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:16.588318 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:16.588327 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:16.588821 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:16.588825 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:16.589014 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:16.589575 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:16.589591 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:16.591320 (+ 1729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:13:16.593393 (+ 2073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:16.593398 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:16.593416 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:16.593710 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:16.593988 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:16.593992 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:16.593994 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:16.594215 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:16.594598 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:13:16.594604 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:13:16.594946 (+ 342us) client_negotiation.cc:769] Sending connection context
0620 19:13:16.595154 (+ 208us) client_negotiation.cc:240] Negotiation successful
0620 19:13:16.595480 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":114,"threads_started":1}
I20260620 19:13:16.597448 26223 heartbeater.cc:347] Connected to a master server at 127.9.155.126:40693
I20260620 19:13:16.597671 26223 heartbeater.cc:464] Registering TS with master...
I20260620 19:13:16.598121 26223 heartbeater.cc:511] Master 127.9.155.126:40693 requested a full tablet report, sending...
I20260620 19:13:16.599061 25765 ts_manager.cc:194] Registered new tserver with Master: 0b46980251d54a8bb398e0da234a85ec (127.9.155.67:38157)
I20260620 19:13:16.599583 25765 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.155.67@KRBTEST.COM'} at 127.9.155.67:41233
I20260620 19:13:16.609356 9837 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:13:16.613564 25955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:16.610048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51366 (local address 127.9.155.126:40693)
0620 19:13:16.610167 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:16.610170 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:16.610303 (+ 133us) server_negotiation.cc:407] Connection header received
0620 19:13:16.610456 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:16.610459 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:16.610587 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:16.610677 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:13:16.610932 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:16.611564 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:16.612476 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:16.612729 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:16.612804 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:16.613048 (+ 244us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:13:16.613197 (+ 149us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0620 19:13:16.613412 (+ 215us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51366: token expired
Metrics: {"server-negotiator.queue_time_us":36}
W20260620 19:13:16.613633 25955 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51366: token expired
W20260620 19:13:16.613965 25685 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.9.155.126:40693)
I20260620 19:13:16.620684 25955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:16.614309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51370 (local address 127.9.155.126:40693)
0620 19:13:16.614411 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:16.614415 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:16.614427 (+ 12us) server_negotiation.cc:407] Connection header received
0620 19:13:16.614494 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:16.614497 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:16.614539 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:16.614601 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:16.615169 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:16.615669 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:16.616461 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:16.616664 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:16.616734 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:16.617955 (+ 1221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:13:16.617958 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:13:16.617959 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:13:16.617977 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:13:16.619280 (+ 1303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:16.619636 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:16.619639 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:16.619640 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:16.619705 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:16.619880 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:16.619882 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:16.619883 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:16.620055 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:13:16.620151 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:16.620454 (+ 303us) server_negotiation.cc:299] Negotiation successful
0620 19:13:16.620552 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 19:13:16.623282 25765 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51370:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:13:16.626710 25765 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:13:16.635106 25887 tablet_service.cc:1467] Processing CreateTablet for tablet 463e435ef88445068b9a83b60d90c288 (DEFAULT_TABLE table=test-table [id=50ce096bf95a44698e0012bb17804e39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:13:16.635787 25887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 463e435ef88445068b9a83b60d90c288. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:13:16.638829 25845 tablet_bootstrap.cc:492] T 463e435ef88445068b9a83b60d90c288 P 2dee97c110914202b2e9ca8db83df0bf: Bootstrap starting.
I20260620 19:13:16.639134 25960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:16.633599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.66:45875 (local address 127.0.0.1:44878)
0620 19:13:16.633765 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:16.633777 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:16.633842 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:16.634826 (+ 984us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:16.634828 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:16.634844 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:13:16.634959 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:16.634965 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:16.636273 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:16.636277 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:16.637464 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:16.637476 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:16.638687 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:16.638690 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:16.638794 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:16.638848 (+ 54us) client_negotiation.cc:769] Sending connection context
0620 19:13:16.638904 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 19:13:16.638998 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94}
I20260620 19:13:16.639689 26232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:16.633897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.67:38157 (local address 127.0.0.1:49730)
0620 19:13:16.634531 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:16.634542 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:16.634628 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:16.635065 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:16.635068 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:16.635081 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:13:16.635172 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:16.635177 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:16.636542 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:16.636545 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:16.637447 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:16.637458 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:16.638630 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:16.638634 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:16.639372 (+ 738us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:16.639402 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 19:13:16.639486 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 19:13:16.639563 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"mutex_wait_us":5,"thread_start_us":103,"threads_started":1}
I20260620 19:13:16.639777 26233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:16.634023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44878 (local address 127.9.155.66:45875)
0620 19:13:16.634313 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:16.634326 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:16.634351 (+ 25us) server_negotiation.cc:407] Connection header received
0620 19:13:16.634427 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:16.634432 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:16.634683 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:16.634852 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:13:16.635039 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:16.636108 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:16.637621 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:16.638559 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:16.639322 (+ 763us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:16.639486 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:16.639542 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 19:13:16.639638 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":99,"threads_started":1}
I20260620 19:13:16.639832 25845 tablet_bootstrap.cc:654] T 463e435ef88445068b9a83b60d90c288 P 2dee97c110914202b2e9ca8db83df0bf: Neither blocks nor log segments found. Creating new log.
I20260620 19:13:16.640564 26234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:16.634124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49730 (local address 127.9.155.67:38157)
0620 19:13:16.634455 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:16.634466 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:16.634625 (+ 159us) server_negotiation.cc:407] Connection header received
0620 19:13:16.634730 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:16.634734 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:16.634912 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:16.635063 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:13:16.635254 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:16.636385 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:16.637576 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:16.638456 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:16.638572 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:16.638733 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:16.640357 (+ 1624us) server_negotiation.cc:299] Negotiation successful
0620 19:13:16.640446 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":146,"threads_started":1}
I20260620 19:13:16.641543 25845 tablet_bootstrap.cc:492] T 463e435ef88445068b9a83b60d90c288 P 2dee97c110914202b2e9ca8db83df0bf: No bootstrap required, opened a new log
I20260620 19:13:16.641646 25845 ts_tablet_manager.cc:1397] T 463e435ef88445068b9a83b60d90c288 P 2dee97c110914202b2e9ca8db83df0bf: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260620 19:13:16.642104 25845 raft_consensus.cc:348] T 463e435ef88445068b9a83b60d90c288 P 2dee97c110914202b2e9ca8db83df0bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a830c89ba25b4f8cb3b47d173e20ff3e" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 45875 } } peers { permanent_uuid: "0b46980251d54a8bb398e0da234a85ec" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 38157 } } peers { permanent_uuid: "2dee97c110914202b2e9ca8db83df0bf" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 39169 } }
I20260620 19:13:16.642220 25845 raft_consensus.cc:374] T 463e435ef88445068b9a83b60d90c288 P 2dee97c110914202b2e9ca8db83df0bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:13:16.642256 25845 raft_consensus.cc:729] T 463e435ef88445068b9a83b60d90c288 P 2dee97c110914202b2e9ca8db83df0bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dee97c110914202b2e9ca8db83df0bf, State: Initialized, Role: FOLLOWER
I20260620 19:13:16.642386 25845 consensus_queue.cc:260] T 463e435ef88445068b9a83b60d90c288 P 2dee97c110914202b2e9ca8db83df0bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a830c89ba25b4f8cb3b47d173e20ff3e" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 45875 } } peers { permanent_uuid: "0b46980251d54a8bb398e0da234a85ec" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 38157 } } peers { permanent_uuid: "2dee97c110914202b2e9ca8db83df0bf" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 39169 } }
I20260620 19:13:16.642654 25845 ts_tablet_manager.cc:1428] T 463e435ef88445068b9a83b60d90c288 P 2dee97c110914202b2e9ca8db83df0bf: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 19:13:16.642985 26025 tablet_service.cc:1467] Processing CreateTablet for tablet 463e435ef88445068b9a83b60d90c288 (DEFAULT_TABLE table=test-table [id=50ce096bf95a44698e0012bb17804e39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:13:16.643431 26157 tablet_service.cc:1467] Processing CreateTablet for tablet 463e435ef88445068b9a83b60d90c288 (DEFAULT_TABLE table=test-table [id=50ce096bf95a44698e0012bb17804e39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:13:16.644181 26025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 463e435ef88445068b9a83b60d90c288. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:13:16.644409 26157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 463e435ef88445068b9a83b60d90c288. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:13:16.651208 26236 tablet_bootstrap.cc:492] T 463e435ef88445068b9a83b60d90c288 P 0b46980251d54a8bb398e0da234a85ec: Bootstrap starting.
I20260620 19:13:16.651752 26237 tablet_bootstrap.cc:492] T 463e435ef88445068b9a83b60d90c288 P a830c89ba25b4f8cb3b47d173e20ff3e: Bootstrap starting.
I20260620 19:13:16.653688 26236 tablet_bootstrap.cc:654] T 463e435ef88445068b9a83b60d90c288 P 0b46980251d54a8bb398e0da234a85ec: Neither blocks nor log segments found. Creating new log.
I20260620 19:13:16.654172 26237 tablet_bootstrap.cc:654] T 463e435ef88445068b9a83b60d90c288 P a830c89ba25b4f8cb3b47d173e20ff3e: Neither blocks nor log segments found. Creating new log.
I20260620 19:13:16.654999 26236 log.cc:824] T 463e435ef88445068b9a83b60d90c288 P 0b46980251d54a8bb398e0da234a85ec: Log is configured to *not* fsync() on all Append() calls
I20260620 19:13:16.654995 26237 log.cc:824] T 463e435ef88445068b9a83b60d90c288 P a830c89ba25b4f8cb3b47d173e20ff3e: Log is configured to *not* fsync() on all Append() calls
I20260620 19:13:16.656973 26237 tablet_bootstrap.cc:492] T 463e435ef88445068b9a83b60d90c288 P a830c89ba25b4f8cb3b47d173e20ff3e: No bootstrap required, opened a new log
I20260620 19:13:16.656983 26236 tablet_bootstrap.cc:492] T 463e435ef88445068b9a83b60d90c288 P 0b46980251d54a8bb398e0da234a85ec: No bootstrap required, opened a new log
I20260620 19:13:16.657155 26237 ts_tablet_manager.cc:1397] T 463e435ef88445068b9a83b60d90c288 P a830c89ba25b4f8cb3b47d173e20ff3e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 19:13:16.657157 26236 ts_tablet_manager.cc:1397] T 463e435ef88445068b9a83b60d90c288 P 0b46980251d54a8bb398e0da234a85ec: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260620 19:13:16.659996 25959 raft_consensus.cc:482] T 463e435ef88445068b9a83b60d90c288 P 2dee97c110914202b2e9ca8db83df0bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:13:16.660192 25959 raft_consensus.cc:504] T 463e435ef88445068b9a83b60d90c288 P 2dee97c110914202b2e9ca8db83df0bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a830c89ba25b4f8cb3b47d173e20ff3e" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 45875 } } peers { permanent_uuid: "0b46980251d54a8bb398e0da234a85ec" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 38157 } } peers { permanent_uuid: "2dee97c110914202b2e9ca8db83df0bf" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 39169 } }
I20260620 19:13:16.660140 26237 raft_consensus.cc:348] T 463e435ef88445068b9a83b60d90c288 P a830c89ba25b4f8cb3b47d173e20ff3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a830c89ba25b4f8cb3b47d173e20ff3e" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 45875 } } peers { permanent_uuid: "0b46980251d54a8bb398e0da234a85ec" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 38157 } } peers { permanent_uuid: "2dee97c110914202b2e9ca8db83df0bf" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 39169 } }
I20260620 19:13:16.660451 26237 raft_consensus.cc:374] T 463e435ef88445068b9a83b60d90c288 P a830c89ba25b4f8cb3b47d173e20ff3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:13:16.660518 26237 raft_consensus.cc:729] T 463e435ef88445068b9a83b60d90c288 P a830c89ba25b4f8cb3b47d173e20ff3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a830c89ba25b4f8cb3b47d173e20ff3e, State: Initialized, Role: FOLLOWER
I20260620 19:13:16.660908 26237 consensus_queue.cc:260] T 463e435ef88445068b9a83b60d90c288 P a830c89ba25b4f8cb3b47d173e20ff3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a830c89ba25b4f8cb3b47d173e20ff3e" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 45875 } } peers { permanent_uuid: "0b46980251d54a8bb398e0da234a85ec" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 38157 } } peers { permanent_uuid: "2dee97c110914202b2e9ca8db83df0bf" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 39169 } }
I20260620 19:13:16.661350 25959 leader_election.cc:288] T 463e435ef88445068b9a83b60d90c288 P 2dee97c110914202b2e9ca8db83df0bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a830c89ba25b4f8cb3b47d173e20ff3e (127.9.155.66:45875), 0b46980251d54a8bb398e0da234a85ec (127.9.155.67:38157)
I20260620 19:13:16.661605 26091 heartbeater.cc:503] Master 127.9.155.126:40693 was elected leader, sending a full tablet report...
I20260620 19:13:16.661913 26237 ts_tablet_manager.cc:1428] T 463e435ef88445068b9a83b60d90c288 P a830c89ba25b4f8cb3b47d173e20ff3e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 19:13:16.662369 26236 raft_consensus.cc:348] T 463e435ef88445068b9a83b60d90c288 P 0b46980251d54a8bb398e0da234a85ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a830c89ba25b4f8cb3b47d173e20ff3e" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 45875 } } peers { permanent_uuid: "0b46980251d54a8bb398e0da234a85ec" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 38157 } } peers { permanent_uuid: "2dee97c110914202b2e9ca8db83df0bf" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 39169 } }
I20260620 19:13:16.662643 26236 raft_consensus.cc:374] T 463e435ef88445068b9a83b60d90c288 P 0b46980251d54a8bb398e0da234a85ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:13:16.662714 26236 raft_consensus.cc:729] T 463e435ef88445068b9a83b60d90c288 P 0b46980251d54a8bb398e0da234a85ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b46980251d54a8bb398e0da234a85ec, State: Initialized, Role: FOLLOWER
I20260620 19:13:16.663125 26236 consensus_queue.cc:260] T 463e435ef88445068b9a83b60d90c288 P 0b46980251d54a8bb398e0da234a85ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a830c89ba25b4f8cb3b47d173e20ff3e" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 45875 } } peers { permanent_uuid: "0b46980251d54a8bb398e0da234a85ec" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 38157 } } peers { permanent_uuid: "2dee97c110914202b2e9ca8db83df0bf" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 39169 } }
I20260620 19:13:16.664059 26236 ts_tablet_manager.cc:1428] T 463e435ef88445068b9a83b60d90c288 P 0b46980251d54a8bb398e0da234a85ec: Time spent starting tablet: real 0.007s user 0.004s sys 0.001s
I20260620 19:13:16.665059 26223 heartbeater.cc:503] Master 127.9.155.126:40693 was elected leader, sending a full tablet report...
I20260620 19:13:16.667055 26233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:16.662079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.65:42291 (local address 127.9.155.66:45875)
0620 19:13:16.662197 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:16.662200 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:16.662210 (+ 10us) server_negotiation.cc:407] Connection header received
0620 19:13:16.662247 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:16.662249 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:16.662282 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:16.662348 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:13:16.662822 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:16.663504 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:16.665861 (+ 2357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:16.666371 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:16.666498 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:16.666551 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:16.666861 (+ 310us) server_negotiation.cc:299] Negotiation successful
0620 19:13:16.666932 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 19:13:16.667079 25956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:16.661528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.66:45875 (local address 127.9.155.65:42291)
0620 19:13:16.661780 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:16.661795 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:16.661889 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:16.662383 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:16.662386 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:16.662405 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:13:16.662519 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:16.662524 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:16.663636 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:16.663640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:16.664808 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:16.664820 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:16.666615 (+ 1795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:16.666617 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:16.666714 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:16.666774 (+ 60us) client_negotiation.cc:769] Sending connection context
0620 19:13:16.666845 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 19:13:16.666954 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":3}
I20260620 19:13:16.667729 26045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "463e435ef88445068b9a83b60d90c288" candidate_uuid: "2dee97c110914202b2e9ca8db83df0bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a830c89ba25b4f8cb3b47d173e20ff3e" is_pre_election: true
I20260620 19:13:16.667997 26045 raft_consensus.cc:2454] T 463e435ef88445068b9a83b60d90c288 P a830c89ba25b4f8cb3b47d173e20ff3e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2dee97c110914202b2e9ca8db83df0bf in term 0.
I20260620 19:13:16.668318 26241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:16.661659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.67:38157 (local address 127.9.155.65:56819)
0620 19:13:16.664494 (+ 2835us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:16.664506 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:16.664601 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:16.664794 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:16.664797 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:16.664811 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:13:16.664920 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:16.664924 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:16.665630 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:16.665633 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:16.667007 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:16.667018 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:16.667803 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:16.667807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:16.668024 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:16.668047 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 19:13:16.668105 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 19:13:16.668154 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2749,"mutex_wait_us":30,"thread_start_us":88,"threads_started":1}
I20260620 19:13:16.668336 26234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:16.662006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.65:56819 (local address 127.9.155.67:38157)
0620 19:13:16.662108 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:16.662113 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:16.664585 (+ 2472us) server_negotiation.cc:407] Connection header received
0620 19:13:16.664676 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:16.664679 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:16.664720 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:16.664780 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:13:16.665018 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:16.665531 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:16.667116 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:16.667669 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:16.667786 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:16.667832 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:16.668129 (+ 297us) server_negotiation.cc:299] Negotiation successful
0620 19:13:16.668200 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 19:13:16.668507 25841 leader_election.cc:302] T 463e435ef88445068b9a83b60d90c288 P 2dee97c110914202b2e9ca8db83df0bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2dee97c110914202b2e9ca8db83df0bf, a830c89ba25b4f8cb3b47d173e20ff3e; no voters:
I20260620 19:13:16.668812 25959 raft_consensus.cc:2781] T 463e435ef88445068b9a83b60d90c288 P 2dee97c110914202b2e9ca8db83df0bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:13:16.668915 25959 raft_consensus.cc:482] T 463e435ef88445068b9a83b60d90c288 P 2dee97c110914202b2e9ca8db83df0bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:13:16.668962 25959 raft_consensus.cc:3037] T 463e435ef88445068b9a83b60d90c288 P 2dee97c110914202b2e9ca8db83df0bf [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:13:16.668947 26177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "463e435ef88445068b9a83b60d90c288" candidate_uuid: "2dee97c110914202b2e9ca8db83df0bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b46980251d54a8bb398e0da234a85ec" is_pre_election: true
I20260620 19:13:16.669185 26177 raft_consensus.cc:2454] T 463e435ef88445068b9a83b60d90c288 P 0b46980251d54a8bb398e0da234a85ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2dee97c110914202b2e9ca8db83df0bf in term 0.
I20260620 19:13:16.670143 25959 raft_consensus.cc:504] T 463e435ef88445068b9a83b60d90c288 P 2dee97c110914202b2e9ca8db83df0bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a830c89ba25b4f8cb3b47d173e20ff3e" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 45875 } } peers { permanent_uuid: "0b46980251d54a8bb398e0da234a85ec" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 38157 } } peers { permanent_uuid: "2dee97c110914202b2e9ca8db83df0bf" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 39169 } }
I20260620 19:13:16.670444 25959 leader_election.cc:288] T 463e435ef88445068b9a83b60d90c288 P 2dee97c110914202b2e9ca8db83df0bf [CANDIDATE]: Term 1 election: Requested vote from peers a830c89ba25b4f8cb3b47d173e20ff3e (127.9.155.66:45875), 0b46980251d54a8bb398e0da234a85ec (127.9.155.67:38157)
I20260620 19:13:16.670775 26177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "463e435ef88445068b9a83b60d90c288" candidate_uuid: "2dee97c110914202b2e9ca8db83df0bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b46980251d54a8bb398e0da234a85ec"
I20260620 19:13:16.670806 26045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "463e435ef88445068b9a83b60d90c288" candidate_uuid: "2dee97c110914202b2e9ca8db83df0bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a830c89ba25b4f8cb3b47d173e20ff3e"
I20260620 19:13:16.670868 26177 raft_consensus.cc:3037] T 463e435ef88445068b9a83b60d90c288 P 0b46980251d54a8bb398e0da234a85ec [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:13:16.670917 26045 raft_consensus.cc:3037] T 463e435ef88445068b9a83b60d90c288 P a830c89ba25b4f8cb3b47d173e20ff3e [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:13:16.671788 26177 raft_consensus.cc:2454] T 463e435ef88445068b9a83b60d90c288 P 0b46980251d54a8bb398e0da234a85ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2dee97c110914202b2e9ca8db83df0bf in term 1.
I20260620 19:13:16.671936 26045 raft_consensus.cc:2454] T 463e435ef88445068b9a83b60d90c288 P a830c89ba25b4f8cb3b47d173e20ff3e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2dee97c110914202b2e9ca8db83df0bf in term 1.
I20260620 19:13:16.672068 25840 leader_election.cc:302] T 463e435ef88445068b9a83b60d90c288 P 2dee97c110914202b2e9ca8db83df0bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b46980251d54a8bb398e0da234a85ec, 2dee97c110914202b2e9ca8db83df0bf; no voters:
I20260620 19:13:16.672257 25959 raft_consensus.cc:2781] T 463e435ef88445068b9a83b60d90c288 P 2dee97c110914202b2e9ca8db83df0bf [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:13:16.672503 25959 raft_consensus.cc:686] T 463e435ef88445068b9a83b60d90c288 P 2dee97c110914202b2e9ca8db83df0bf [term 1 LEADER]: Becoming Leader. State: Replica: 2dee97c110914202b2e9ca8db83df0bf, State: Running, Role: LEADER
I20260620 19:13:16.672755 25959 consensus_queue.cc:237] T 463e435ef88445068b9a83b60d90c288 P 2dee97c110914202b2e9ca8db83df0bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a830c89ba25b4f8cb3b47d173e20ff3e" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 45875 } } peers { permanent_uuid: "0b46980251d54a8bb398e0da234a85ec" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 38157 } } peers { permanent_uuid: "2dee97c110914202b2e9ca8db83df0bf" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 39169 } }
I20260620 19:13:16.675484 25764 catalog_manager.cc:5495] T 463e435ef88445068b9a83b60d90c288 P 2dee97c110914202b2e9ca8db83df0bf reported cstate change: term changed from 0 to 1, leader changed from <none> to 2dee97c110914202b2e9ca8db83df0bf (127.9.155.65). New cstate: current_term: 1 leader_uuid: "2dee97c110914202b2e9ca8db83df0bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a830c89ba25b4f8cb3b47d173e20ff3e" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 45875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b46980251d54a8bb398e0da234a85ec" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 38157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2dee97c110914202b2e9ca8db83df0bf" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 39169 } health_report { overall_health: HEALTHY } } }
I20260620 19:13:16.694811 25961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:16.691306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48142 (local address 127.9.155.65:39169)
0620 19:13:16.691432 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:16.691436 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:16.691449 (+ 13us) server_negotiation.cc:407] Connection header received
0620 19:13:16.691575 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:16.691579 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:16.691644 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:16.691736 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:13:16.691902 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:16.692491 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:16.693330 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:16.693591 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:16.693680 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:16.693932 (+ 252us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:13:16.694385 (+ 453us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:13:16.694480 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:16.694592 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 19:13:16.694662 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 19:13:16.698789 26045 raft_consensus.cc:1261] T 463e435ef88445068b9a83b60d90c288 P a830c89ba25b4f8cb3b47d173e20ff3e [term 1 FOLLOWER]: Refusing update from remote peer 2dee97c110914202b2e9ca8db83df0bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:13:16.698791 26177 raft_consensus.cc:1261] T 463e435ef88445068b9a83b60d90c288 P 0b46980251d54a8bb398e0da234a85ec [term 1 FOLLOWER]: Refusing update from remote peer 2dee97c110914202b2e9ca8db83df0bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:13:16.699486 26243 consensus_queue.cc:1035] T 463e435ef88445068b9a83b60d90c288 P 2dee97c110914202b2e9ca8db83df0bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b46980251d54a8bb398e0da234a85ec" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 38157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:13:16.699664 25959 consensus_queue.cc:1035] T 463e435ef88445068b9a83b60d90c288 P 2dee97c110914202b2e9ca8db83df0bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "a830c89ba25b4f8cb3b47d173e20ff3e" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 45875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 19:13:16.704169 25954 tablet.cc:2369] T 463e435ef88445068b9a83b60d90c288 P 2dee97c110914202b2e9ca8db83df0bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:13:16.706640 26248 mvcc.cc:205] Tried to move back new op lower bound from 7299001535271682048 to 7299001535173959680. Current Snapshot: MvccSnapshot[applied={T|T < 7299001535271682048}]
I20260620 19:13:16.708273 26247 mvcc.cc:205] Tried to move back new op lower bound from 7299001535271682048 to 7299001535173959680. Current Snapshot: MvccSnapshot[applied={T|T < 7299001535271682048}]
I20260620 19:13:16.715754 25765 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51370:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:13:16.715947 25765 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51370:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:13:16.718495 25765 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 025239f00c2441fe885affb9769996d9: Sending DeleteTablet for 3 replicas of tablet 463e435ef88445068b9a83b60d90c288
I20260620 19:13:16.719259 26025 tablet_service.cc:1514] Processing DeleteTablet for tablet 463e435ef88445068b9a83b60d90c288 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:13:16 UTC) from {username='kudu', principal='kudu/127.9.155.126@KRBTEST.COM'} at 127.0.0.1:44878
I20260620 19:13:16.719277 26157 tablet_service.cc:1514] Processing DeleteTablet for tablet 463e435ef88445068b9a83b60d90c288 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:13:16 UTC) from {username='kudu', principal='kudu/127.9.155.126@KRBTEST.COM'} at 127.0.0.1:49730
I20260620 19:13:16.719331 25887 tablet_service.cc:1514] Processing DeleteTablet for tablet 463e435ef88445068b9a83b60d90c288 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:13:16 UTC) from {username='kudu', principal='kudu/127.9.155.126@KRBTEST.COM'} at 127.0.0.1:48132
I20260620 19:13:16.719496 25962 tablet_replica.cc:330] T 463e435ef88445068b9a83b60d90c288 P 2dee97c110914202b2e9ca8db83df0bf: stopping tablet replica
I20260620 19:13:16.719606 25962 raft_consensus.cc:2229] T 463e435ef88445068b9a83b60d90c288 P 2dee97c110914202b2e9ca8db83df0bf [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:13:16.719796 26257 tablet_replica.cc:330] T 463e435ef88445068b9a83b60d90c288 P a830c89ba25b4f8cb3b47d173e20ff3e: stopping tablet replica
I20260620 19:13:16.719924 25962 raft_consensus.cc:2258] T 463e435ef88445068b9a83b60d90c288 P 2dee97c110914202b2e9ca8db83df0bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:13:16.719961 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 25803
I20260620 19:13:16.720131 26257 raft_consensus.cc:2229] T 463e435ef88445068b9a83b60d90c288 P a830c89ba25b4f8cb3b47d173e20ff3e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:13:16.720609 26257 raft_consensus.cc:2258] T 463e435ef88445068b9a83b60d90c288 P a830c89ba25b4f8cb3b47d173e20ff3e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:13:16.720981 26256 tablet_replica.cc:330] T 463e435ef88445068b9a83b60d90c288 P 0b46980251d54a8bb398e0da234a85ec: stopping tablet replica
I20260620 19:13:16.721279 26256 raft_consensus.cc:2229] T 463e435ef88445068b9a83b60d90c288 P 0b46980251d54a8bb398e0da234a85ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:13:16.721570 26256 raft_consensus.cc:2258] T 463e435ef88445068b9a83b60d90c288 P 0b46980251d54a8bb398e0da234a85ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:13:16.722564 26257 ts_tablet_manager.cc:1905] T 463e435ef88445068b9a83b60d90c288 P a830c89ba25b4f8cb3b47d173e20ff3e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:13:16.722977 26256 ts_tablet_manager.cc:1905] T 463e435ef88445068b9a83b60d90c288 P 0b46980251d54a8bb398e0da234a85ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:13:16.725275 26257 ts_tablet_manager.cc:1918] T 463e435ef88445068b9a83b60d90c288 P a830c89ba25b4f8cb3b47d173e20ff3e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:13:16.725380 26257 log.cc:1196] T 463e435ef88445068b9a83b60d90c288 P a830c89ba25b4f8cb3b47d173e20ff3e: Deleting WAL directory at /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-1/wal/wals/463e435ef88445068b9a83b60d90c288
I20260620 19:13:16.725404 26256 ts_tablet_manager.cc:1918] T 463e435ef88445068b9a83b60d90c288 P 0b46980251d54a8bb398e0da234a85ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:13:16.725482 26256 log.cc:1196] T 463e435ef88445068b9a83b60d90c288 P 0b46980251d54a8bb398e0da234a85ec: Deleting WAL directory at /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-2/wal/wals/463e435ef88445068b9a83b60d90c288
I20260620 19:13:16.725724 26257 ts_tablet_manager.cc:1939] T 463e435ef88445068b9a83b60d90c288 P a830c89ba25b4f8cb3b47d173e20ff3e: Deleting consensus metadata
I20260620 19:13:16.725772 26256 ts_tablet_manager.cc:1939] T 463e435ef88445068b9a83b60d90c288 P 0b46980251d54a8bb398e0da234a85ec: Deleting consensus metadata
I20260620 19:13:16.727057 25750 catalog_manager.cc:4841] TS 0b46980251d54a8bb398e0da234a85ec (127.9.155.67:38157): tablet 463e435ef88445068b9a83b60d90c288 (table test-table [id=50ce096bf95a44698e0012bb17804e39]) successfully deleted
I20260620 19:13:16.727092 25751 catalog_manager.cc:4841] TS a830c89ba25b4f8cb3b47d173e20ff3e (127.9.155.66:45875): tablet 463e435ef88445068b9a83b60d90c288 (table test-table [id=50ce096bf95a44698e0012bb17804e39]) successfully deleted
W20260620 19:13:16.728272 25749 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.155.65:39169 (error 108)
I20260620 19:13:16.728601 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 25958
I20260620 19:13:16.730521 26232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:16.729933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.65:39169 (local address 127.0.0.1:48156)
0620 19:13:16.730364 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:16.730434 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.65:39169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335}
W20260620 19:13:16.730674 25749 catalog_manager.cc:4584] TS 2dee97c110914202b2e9ca8db83df0bf (127.9.155.65:39169): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 463e435ef88445068b9a83b60d90c288: Network error: Client connection negotiation failed: client connection to 127.9.155.65:39169: connect: Connection refused (error 111)
I20260620 19:13:16.733419 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 26093
I20260620 19:13:16.738176 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 25732
I20260620 19:13:36.744103 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:40693
--webserver_interface=127.9.155.126
--webserver_port=40719
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab
--principal=kudu/127.9.155.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:42519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.155.126:40693
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:13:36.866261 26259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:13:36.866560 26259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:13:36.866602 26259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:13:36.870760 26259 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 19:13:36.870814 26259 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 19:13:36.870829 26259 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 19:13:36.870842 26259 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:13:36.870858 26259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:13:36.870874 26259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:13:36.870889 26259 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:13:36.870908 26259 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:13:36.876276 26259 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:42519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.9.155.126:40693
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab
--principal=kudu/127.9.155.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:40693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.155.126
--webserver_port=40719
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26259
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:13:36.877594 26259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:13:36.878713 26259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:13:36.889219 26265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:13:36.889281 26264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:13:36.889330 26259 server_base.cc:1034] running on GCE node
W20260620 19:13:36.889281 26267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:13:36.890141 26259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:13:36.891510 26259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:13:36.892678 26259 hybrid_clock.cc:648] HybridClock initialized: now 1781982816892648 us; error 46 us; skew 500 ppm
Jun 20 19:13:36 dist-test-slave-n3fx krb5kdc[25178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982816, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:13:36.895833 26259 init.cc:377] Logged in from keytab as kudu/127.9.155.126@KRBTEST.COM (short username kudu)
I20260620 19:13:36.896912 26259 webserver.cc:460] Webserver started at http://127.9.155.126:40719/ using document root <none> and password file <none>
I20260620 19:13:36.897523 26259 fs_manager.cc:359] Metadata directory not provided
I20260620 19:13:36.897583 26259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:13:36.902087 26259 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 19:13:36.904578 26274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:36.906109 26259 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:13:36.906221 26259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/master-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/master-0/wal
uuid: "025239f00c2441fe885affb9769996d9"
format_stamp: "Formatted at 2026-06-20 19:12:54 on dist-test-slave-n3fx"
server_key: "0c722aadc3be79eeb06f6000b03a0746"
server_key_iv: "a9c7db7ad71d08c9f106596a6f7c0434"
server_key_version: "encryptionkey@0"
I20260620 19:13:36.906843 26259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:13:36.942824 26259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:13:36.946460 26259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:13:36.946655 26259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:13:36.954632 26259 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.126:40693
I20260620 19:13:36.954950 26326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.126:40693 every 8 connection(s)
I20260620 19:13:36.955835 26259 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
I20260620 19:13:36.960992 26327 sys_catalog.cc:263] Verifying existing consensus state
I20260620 19:13:36.961808 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 26259
I20260620 19:13:36.962286 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.65:39169
--local_ip_for_outbound_sockets=127.9.155.65
--tserver_master_addrs=127.9.155.126:40693
--webserver_port=36173
--webserver_interface=127.9.155.65
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.65.keytab
--principal=kudu/127.9.155.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:42519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:13:36.962977 26327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 025239f00c2441fe885affb9769996d9: Bootstrap starting.
I20260620 19:13:36.982662 26327 log.cc:824] T 00000000000000000000000000000000 P 025239f00c2441fe885affb9769996d9: Log is configured to *not* fsync() on all Append() calls
I20260620 19:13:36.994743 26327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 025239f00c2441fe885affb9769996d9: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:13:36.995146 26327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 025239f00c2441fe885affb9769996d9: Bootstrap complete.
I20260620 19:13:36.999701 26327 raft_consensus.cc:348] T 00000000000000000000000000000000 P 025239f00c2441fe885affb9769996d9 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "025239f00c2441fe885affb9769996d9" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 40693 } }
I20260620 19:13:37.000144 26327 raft_consensus.cc:729] T 00000000000000000000000000000000 P 025239f00c2441fe885affb9769996d9 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 025239f00c2441fe885affb9769996d9, State: Initialized, Role: FOLLOWER
I20260620 19:13:37.000753 26327 consensus_queue.cc:260] T 00000000000000000000000000000000 P 025239f00c2441fe885affb9769996d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "025239f00c2441fe885affb9769996d9" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 40693 } }
I20260620 19:13:37.000896 26327 raft_consensus.cc:388] T 00000000000000000000000000000000 P 025239f00c2441fe885affb9769996d9 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:13:37.000970 26327 raft_consensus.cc:482] T 00000000000000000000000000000000 P 025239f00c2441fe885affb9769996d9 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:13:37.001072 26327 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 025239f00c2441fe885affb9769996d9 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:13:37.003226 26327 raft_consensus.cc:504] T 00000000000000000000000000000000 P 025239f00c2441fe885affb9769996d9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "025239f00c2441fe885affb9769996d9" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 40693 } }
I20260620 19:13:37.003594 26327 leader_election.cc:302] T 00000000000000000000000000000000 P 025239f00c2441fe885affb9769996d9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 025239f00c2441fe885affb9769996d9; no voters:
I20260620 19:13:37.003940 26327 leader_election.cc:288] T 00000000000000000000000000000000 P 025239f00c2441fe885affb9769996d9 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 19:13:37.004063 26331 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 025239f00c2441fe885affb9769996d9 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 19:13:37.004297 26331 raft_consensus.cc:686] T 00000000000000000000000000000000 P 025239f00c2441fe885affb9769996d9 [term 3 LEADER]: Becoming Leader. State: Replica: 025239f00c2441fe885affb9769996d9, State: Running, Role: LEADER
I20260620 19:13:37.004629 26331 consensus_queue.cc:237] T 00000000000000000000000000000000 P 025239f00c2441fe885affb9769996d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "025239f00c2441fe885affb9769996d9" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 40693 } }
I20260620 19:13:37.005380 26327 sys_catalog.cc:564] T 00000000000000000000000000000000 P 025239f00c2441fe885affb9769996d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:13:37.006311 26333 sys_catalog.cc:455] T 00000000000000000000000000000000 P 025239f00c2441fe885affb9769996d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 025239f00c2441fe885affb9769996d9. Latest consensus state: current_term: 3 leader_uuid: "025239f00c2441fe885affb9769996d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "025239f00c2441fe885affb9769996d9" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 40693 } } }
I20260620 19:13:37.006461 26333 sys_catalog.cc:458] T 00000000000000000000000000000000 P 025239f00c2441fe885affb9769996d9 [sys.catalog]: This master's current role is: LEADER
I20260620 19:13:37.006726 26332 sys_catalog.cc:455] T 00000000000000000000000000000000 P 025239f00c2441fe885affb9769996d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "025239f00c2441fe885affb9769996d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "025239f00c2441fe885affb9769996d9" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 40693 } } }
I20260620 19:13:37.006826 26332 sys_catalog.cc:458] T 00000000000000000000000000000000 P 025239f00c2441fe885affb9769996d9 [sys.catalog]: This master's current role is: LEADER
I20260620 19:13:37.008323 26339 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:13:37.013486 26339 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:13:37.014015 26339 catalog_manager.cc:1259] Loaded cluster ID: 36f418d264f64c6fb041fa4533bace46
I20260620 19:13:37.014063 26339 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:13:37.015936 26339 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:13:37.016561 26339 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 025239f00c2441fe885affb9769996d9: Loaded TSK: 0
I20260620 19:13:37.016623 26339 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 025239f00c2441fe885affb9769996d9: Loaded TSK: 1
F20260620 19:13:37.017287 26339 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fc6016df9ad google::LogMessage::Fail() at ??:0
@ 0x7fc6016e288a google::LogMessage::SendToLog() at ??:0
@ 0x7fc6016df4de google::LogMessage::Flush() at ??:0
@ 0x7fc6016e0a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fc6043a318d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fc60439bf0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fc60c0447bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fc60c01e99c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fc60c01f178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7fc60c060df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fc60c07f82b std::function<>::operator()() at ??:0
@ 0x7fc60c01eb4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fc60c01ff3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fc60c01c579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fc60c060640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fc6039313d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fc603931c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fc6039333f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fc603922512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fc604f666db start_thread at ??:0
@ 0x7fc6001dd61f clone at ??:0
W20260620 19:13:37.107021 26329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:13:37.107288 26329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:13:37.107331 26329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:13:37.111505 26329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:13:37.111560 26329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:13:37.111639 26329 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.65
I20260620 19:13:37.119046 26329 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:42519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.65.keytab
--principal=kudu/127.9.155.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.65:39169
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.155.65
--webserver_port=36173
--tserver_master_addrs=127.9.155.126:40693
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26329
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.65
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:13:37.120496 26329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:13:37.121754 26329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:13:37.133766 26374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:13:37.133800 26375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:13:37.133800 26377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:13:37.134222 26329 server_base.cc:1034] running on GCE node
I20260620 19:13:37.134652 26329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:13:37.135419 26329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:13:37.136587 26329 hybrid_clock.cc:648] HybridClock initialized: now 1781982817136564 us; error 39 us; skew 500 ppm
Jun 20 19:13:37 dist-test-slave-n3fx krb5kdc[25178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982817, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:13:37.139559 26329 init.cc:377] Logged in from keytab as kudu/127.9.155.65@KRBTEST.COM (short username kudu)
I20260620 19:13:37.140558 26329 webserver.cc:460] Webserver started at http://127.9.155.65:36173/ using document root <none> and password file <none>
I20260620 19:13:37.141213 26329 fs_manager.cc:359] Metadata directory not provided
I20260620 19:13:37.141281 26329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:13:37.145553 26329 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 19:13:37.147842 26384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:37.149358 26329 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:13:37.149468 26329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-0/wal
uuid: "2dee97c110914202b2e9ca8db83df0bf"
format_stamp: "Formatted at 2026-06-20 19:12:54 on dist-test-slave-n3fx"
server_key: "6c811a05433d6736303f0a3060afa774"
server_key_iv: "d75bd652d3fdee25dbbd0add8a9744a9"
server_key_version: "encryptionkey@0"
I20260620 19:13:37.150094 26329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:13:37.203150 26329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:13:37.206979 26329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:13:37.207195 26329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:13:37.207916 26329 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:13:37.209308 26391 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 19:13:37.215936 26329 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 19:13:37.216053 26329 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20260620 19:13:37.216135 26329 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 19:13:37.218181 26329 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 19:13:37.218230 26329 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260620 19:13:37.218636 26391 tablet_bootstrap.cc:492] T 463e435ef88445068b9a83b60d90c288 P 2dee97c110914202b2e9ca8db83df0bf: Bootstrap starting.
I20260620 19:13:37.229414 26329 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.65:39169
I20260620 19:13:37.229471 26498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.65:39169 every 8 connection(s)
I20260620 19:13:37.230516 26329 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
I20260620 19:13:37.240658 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 26329
I20260620 19:13:37.241159 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.66:45875
--local_ip_for_outbound_sockets=127.9.155.66
--tserver_master_addrs=127.9.155.126:40693
--webserver_port=45103
--webserver_interface=127.9.155.66
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.66.keytab
--principal=kudu/127.9.155.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:42519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:13:37.243901 26391 log.cc:824] T 463e435ef88445068b9a83b60d90c288 P 2dee97c110914202b2e9ca8db83df0bf: Log is configured to *not* fsync() on all Append() calls
I20260620 19:13:37.254719 26391 tablet_bootstrap.cc:492] T 463e435ef88445068b9a83b60d90c288 P 2dee97c110914202b2e9ca8db83df0bf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:13:37.255136 26391 tablet_bootstrap.cc:492] T 463e435ef88445068b9a83b60d90c288 P 2dee97c110914202b2e9ca8db83df0bf: Bootstrap complete.
I20260620 19:13:37.256469 26391 ts_tablet_manager.cc:1397] T 463e435ef88445068b9a83b60d90c288 P 2dee97c110914202b2e9ca8db83df0bf: Time spent bootstrapping tablet: real 0.038s user 0.021s sys 0.016s
I20260620 19:13:37.260152 26391 raft_consensus.cc:348] T 463e435ef88445068b9a83b60d90c288 P 2dee97c110914202b2e9ca8db83df0bf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a830c89ba25b4f8cb3b47d173e20ff3e" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 45875 } } peers { permanent_uuid: "0b46980251d54a8bb398e0da234a85ec" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 38157 } } peers { permanent_uuid: "2dee97c110914202b2e9ca8db83df0bf" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 39169 } }
I20260620 19:13:37.260787 26391 raft_consensus.cc:729] T 463e435ef88445068b9a83b60d90c288 P 2dee97c110914202b2e9ca8db83df0bf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dee97c110914202b2e9ca8db83df0bf, State: Initialized, Role: FOLLOWER
I20260620 19:13:37.261406 26391 consensus_queue.cc:260] T 463e435ef88445068b9a83b60d90c288 P 2dee97c110914202b2e9ca8db83df0bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a830c89ba25b4f8cb3b47d173e20ff3e" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 45875 } } peers { permanent_uuid: "0b46980251d54a8bb398e0da234a85ec" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 38157 } } peers { permanent_uuid: "2dee97c110914202b2e9ca8db83df0bf" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 39169 } }
I20260620 19:13:37.262634 26391 ts_tablet_manager.cc:1428] T 463e435ef88445068b9a83b60d90c288 P 2dee97c110914202b2e9ca8db83df0bf: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260620 19:13:37.280539 26501 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:13:37.232686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40693 (local address 127.9.155.65:52939)
0620 19:13:37.233240 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:37.233297 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:37.234293 (+ 996us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:37.279645 (+ 45352us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:40693: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":125,"threads_started":1}
W20260620 19:13:37.281086 26499 heartbeater.cc:644] Failed to heartbeat to 127.9.155.126:40693 (0 consecutive failures): Network error: Failed to ping master at 127.9.155.126:40693: Client connection negotiation failed: client connection to 127.9.155.126:40693: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 19:13:37.282274 26501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:37.281657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40693 (local address 127.9.155.65:38187)
0620 19:13:37.281951 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:37.282097 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
I20260620 19:13:37.283028 26501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:37.282811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40693 (local address 127.9.155.65:38519)
0620 19:13:37.282924 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:37.282973 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 19:13:37.283224 26499 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 19:13:37.369524 26502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:13:37.369805 26502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:13:37.369850 26502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:13:37.373857 26502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:13:37.373911 26502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:13:37.373967 26502 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.66
I20260620 19:13:37.379257 26502 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:42519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.66.keytab
--principal=kudu/127.9.155.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.66:45875
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.155.66
--webserver_port=45103
--tserver_master_addrs=127.9.155.126:40693
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26502
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.66
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:13:37.380429 26502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:13:37.381587 26502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:13:37.392354 26510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:13:37.392527 26502 server_base.cc:1034] running on GCE node
W20260620 19:13:37.392393 26509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:13:37.392354 26512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:13:37.393100 26502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:13:37.393800 26502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:13:37.394977 26502 hybrid_clock.cc:648] HybridClock initialized: now 1781982817394961 us; error 29 us; skew 500 ppm
Jun 20 19:13:37 dist-test-slave-n3fx krb5kdc[25178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982817, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:13:37.398089 26502 init.cc:377] Logged in from keytab as kudu/127.9.155.66@KRBTEST.COM (short username kudu)
I20260620 19:13:37.399089 26502 webserver.cc:460] Webserver started at http://127.9.155.66:45103/ using document root <none> and password file <none>
I20260620 19:13:37.399734 26502 fs_manager.cc:359] Metadata directory not provided
I20260620 19:13:37.399811 26502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:13:37.404490 26502 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 19:13:37.406890 26519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:37.408418 26502 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:13:37.408543 26502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-1/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-1/wal
uuid: "a830c89ba25b4f8cb3b47d173e20ff3e"
format_stamp: "Formatted at 2026-06-20 19:12:54 on dist-test-slave-n3fx"
server_key: "5443a0cb3a88e72d78fd8b772e429b51"
server_key_iv: "6d83aac77ad59a26c37f02f9f4b7f1bc"
server_key_version: "encryptionkey@0"
I20260620 19:13:37.409174 26502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:13:37.425238 26502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:13:37.428536 26502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:13:37.428716 26502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:13:37.429360 26502 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:13:37.430259 26502 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:13:37.430308 26502 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:37.430362 26502 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:13:37.430388 26502 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:37.439918 26502 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.66:45875
I20260620 19:13:37.439970 26632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.66:45875 every 8 connection(s)
I20260620 19:13:37.441041 26502 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
I20260620 19:13:37.445533 26635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:37.443449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40693 (local address 127.9.155.66:53799)
0620 19:13:37.443988 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:37.444116 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":120,"threads_started":1}
W20260620 19:13:37.446033 26633 heartbeater.cc:644] Failed to heartbeat to 127.9.155.126:40693 (0 consecutive failures): Network error: Failed to ping master at 127.9.155.126:40693: Client connection negotiation failed: client connection to 127.9.155.126:40693: connect: Connection refused (error 111)
I20260620 19:13:37.446725 26635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:37.446474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40693 (local address 127.9.155.66:59981)
0620 19:13:37.446605 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:37.446666 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 19:13:37.447352 26635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:37.447147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40693 (local address 127.9.155.66:47121)
0620 19:13:37.447256 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:37.447299 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 19:13:37.447513 26633 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 19:13:37.448760 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 26502
I20260620 19:13:37.449236 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.67:38157
--local_ip_for_outbound_sockets=127.9.155.67
--tserver_master_addrs=127.9.155.126:40693
--webserver_port=45161
--webserver_interface=127.9.155.67
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.67.keytab
--principal=kudu/127.9.155.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:42519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:13:37.570343 26636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:13:37.570629 26636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:13:37.570672 26636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:13:37.574820 26636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:13:37.574906 26636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:13:37.574968 26636 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.67
I20260620 19:13:37.580605 26636 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:42519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.67.keytab
--principal=kudu/127.9.155.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.67:38157
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.155.67
--webserver_port=45161
--tserver_master_addrs=127.9.155.126:40693
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26636
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.67
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:13:37.581847 26636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:13:37.582938 26636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:13:37.593088 26641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:13:37.593082 26642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:13:37.593078 26644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:13:37.593154 26636 server_base.cc:1034] running on GCE node
I20260620 19:13:37.593715 26636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:13:37.594417 26636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:13:37.595577 26636 hybrid_clock.cc:648] HybridClock initialized: now 1781982817595550 us; error 42 us; skew 500 ppm
Jun 20 19:13:37 dist-test-slave-n3fx krb5kdc[25178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982817, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:13:37.598379 26636 init.cc:377] Logged in from keytab as kudu/127.9.155.67@KRBTEST.COM (short username kudu)
I20260620 19:13:37.599284 26636 webserver.cc:460] Webserver started at http://127.9.155.67:45161/ using document root <none> and password file <none>
I20260620 19:13:37.599848 26636 fs_manager.cc:359] Metadata directory not provided
I20260620 19:13:37.599915 26636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:13:37.603951 26636 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:13:37.606246 26651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:37.607589 26636 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 19:13:37.607709 26636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-2/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-2/wal
uuid: "0b46980251d54a8bb398e0da234a85ec"
format_stamp: "Formatted at 2026-06-20 19:12:55 on dist-test-slave-n3fx"
server_key: "d1c948a36f6cce92e06df2322361459e"
server_key_iv: "4557675b576f09102bea8534fc65fb9e"
server_key_version: "encryptionkey@0"
I20260620 19:13:37.608318 26636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:13:37.630329 26636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:13:37.633406 26636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:13:37.633580 26636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:13:37.634186 26636 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:13:37.635022 26636 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:13:37.635069 26636 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:37.635107 26636 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:13:37.635134 26636 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:37.643746 26636 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.67:38157
I20260620 19:13:37.643779 26764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.67:38157 every 8 connection(s)
I20260620 19:13:37.644807 26636 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
I20260620 19:13:37.646311 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 26636
I20260620 19:13:37.648936 26768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:37.647125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40693 (local address 127.9.155.67:52313)
0620 19:13:37.647589 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:37.647695 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":138,"threads_started":1}
W20260620 19:13:37.649398 26765 heartbeater.cc:644] Failed to heartbeat to 127.9.155.126:40693 (0 consecutive failures): Network error: Failed to ping master at 127.9.155.126:40693: Client connection negotiation failed: client connection to 127.9.155.126:40693: connect: Connection refused (error 111)
I20260620 19:13:37.649964 26768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:37.649773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40693 (local address 127.9.155.67:40313)
0620 19:13:37.649865 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:37.649912 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 19:13:37.650569 26768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:37.650388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40693 (local address 127.9.155.67:60001)
0620 19:13:37.650479 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:37.650526 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 19:13:37.650781 26765 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 19:13:38.284471 26769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:38.283938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40693 (local address 127.9.155.65:48629)
0620 19:13:38.284268 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:38.284340 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":79,"threads_started":1}
I20260620 19:13:38.448623 26770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:38.448158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40693 (local address 127.9.155.66:60273)
0620 19:13:38.448445 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:38.448510 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":112,"threads_started":1}
I20260620 19:13:38.651765 26771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:38.651312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40693 (local address 127.9.155.67:51593)
0620 19:13:38.651608 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:38.651663 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":93,"threads_started":1}
I20260620 19:13:39.035388 26772 raft_consensus.cc:482] T 463e435ef88445068b9a83b60d90c288 P 2dee97c110914202b2e9ca8db83df0bf [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:13:39.035580 26772 raft_consensus.cc:504] T 463e435ef88445068b9a83b60d90c288 P 2dee97c110914202b2e9ca8db83df0bf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a830c89ba25b4f8cb3b47d173e20ff3e" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 45875 } } peers { permanent_uuid: "0b46980251d54a8bb398e0da234a85ec" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 38157 } } peers { permanent_uuid: "2dee97c110914202b2e9ca8db83df0bf" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 39169 } }
I20260620 19:13:39.036789 26772 leader_election.cc:288] T 463e435ef88445068b9a83b60d90c288 P 2dee97c110914202b2e9ca8db83df0bf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a830c89ba25b4f8cb3b47d173e20ff3e (127.9.155.66:45875), 0b46980251d54a8bb398e0da234a85ec (127.9.155.67:38157)
Jun 20 19:13:39 dist-test-slave-n3fx krb5kdc[25178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982817, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.65@KRBTEST.COM for kudu/127.9.155.66@KRBTEST.COM
Jun 20 19:13:39 dist-test-slave-n3fx krb5kdc[25178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982817, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.65@KRBTEST.COM for kudu/127.9.155.67@KRBTEST.COM
I20260620 19:13:39.051126 26776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:39.037054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.66:45875 (local address 127.9.155.65:60301)
0620 19:13:39.037730 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:39.037743 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:39.037856 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:39.039171 (+ 1315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:39.039178 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:39.039770 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:39.040298 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:39.040317 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:39.041619 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:39.041622 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:39.042669 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:39.042679 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:39.043399 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:39.043402 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:39.043624 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:39.044317 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:39.044332 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:39.046079 (+ 1747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:13:39.048549 (+ 2470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:39.048553 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:39.048565 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:39.048946 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:39.049300 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:39.049303 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:39.049305 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:39.049465 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:39.050116 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:13:39.050122 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:13:39.050365 (+ 243us) client_negotiation.cc:769] Sending connection context
0620 19:13:39.050537 (+ 172us) client_negotiation.cc:240] Negotiation successful
0620 19:13:39.050915 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"mutex_wait_us":601,"spinlock_wait_cycles":18304,"thread_start_us":83,"threads_started":1}
I20260620 19:13:39.051119 26774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:39.037214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.65:60301 (local address 127.9.155.66:45875)
0620 19:13:39.037528 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:39.037543 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:39.037845 (+ 302us) server_negotiation.cc:407] Connection header received
0620 19:13:39.038502 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:39.038507 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:39.038794 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:39.039095 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:39.040471 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:39.041472 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:39.042795 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:39.043274 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:39.043435 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:39.046319 (+ 2884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:13:39.046324 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:13:39.046338 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:13:39.046369 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:13:39.048437 (+ 2068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:39.049065 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:39.049071 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:39.049076 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:39.049206 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:39.049556 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:39.049559 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:39.049561 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:39.050006 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:13:39.050126 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:39.050666 (+ 540us) server_negotiation.cc:299] Negotiation successful
0620 19:13:39.050935 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":102,"threads_started":1}
I20260620 19:13:39.051120 26775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:39.037232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.65:38393 (local address 127.9.155.67:38157)
0620 19:13:39.037625 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:39.037644 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:39.037680 (+ 36us) server_negotiation.cc:407] Connection header received
0620 19:13:39.038426 (+ 746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:39.038433 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:39.038795 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:39.039072 (+ 277us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:39.040476 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:39.041465 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:39.042801 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:39.043276 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:39.043429 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:39.046623 (+ 3194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:13:39.046628 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:13:39.046640 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:13:39.046690 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:13:39.048407 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:39.049066 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:39.049070 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:39.049075 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:39.049198 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:39.049572 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:39.049575 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:39.049576 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:39.050006 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:13:39.050126 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:39.050671 (+ 545us) server_negotiation.cc:299] Negotiation successful
0620 19:13:39.050930 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":109,"threads_started":1}
I20260620 19:13:39.051172 26773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:39.037054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.67:38157 (local address 127.9.155.65:38393)
0620 19:13:39.037421 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:39.037438 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:39.037548 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:39.039171 (+ 1623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:39.039178 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:39.039770 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:39.040298 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:39.040317 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:39.041623 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:39.041626 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:39.042668 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:39.042678 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:39.043401 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:39.043405 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:39.043625 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:39.044312 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:39.044336 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:39.046416 (+ 2080us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:13:39.048543 (+ 2127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:39.048549 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:39.048564 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:39.048952 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:39.049298 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:39.049302 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:39.049304 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:39.049476 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:39.050117 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:13:39.050123 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:13:39.050365 (+ 242us) client_negotiation.cc:769] Sending connection context
0620 19:13:39.050538 (+ 173us) client_negotiation.cc:240] Negotiation successful
0620 19:13:39.050993 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"mutex_wait_us":283,"spinlock_wait_cycles":71936,"thread_start_us":78,"threads_started":1}
W20260620 19:13:39.053634 26386 leader_election.cc:341] T 463e435ef88445068b9a83b60d90c288 P 2dee97c110914202b2e9ca8db83df0bf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0b46980251d54a8bb398e0da234a85ec (127.9.155.67:38157): Not found: Tablet not found: 463e435ef88445068b9a83b60d90c288
W20260620 19:13:39.053834 26387 leader_election.cc:341] T 463e435ef88445068b9a83b60d90c288 P 2dee97c110914202b2e9ca8db83df0bf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a830c89ba25b4f8cb3b47d173e20ff3e (127.9.155.66:45875): Not found: Tablet not found: 463e435ef88445068b9a83b60d90c288
I20260620 19:13:39.053910 26387 leader_election.cc:302] T 463e435ef88445068b9a83b60d90c288 P 2dee97c110914202b2e9ca8db83df0bf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2dee97c110914202b2e9ca8db83df0bf; no voters: 0b46980251d54a8bb398e0da234a85ec, a830c89ba25b4f8cb3b47d173e20ff3e
I20260620 19:13:39.054126 26772 raft_consensus.cc:2726] T 463e435ef88445068b9a83b60d90c288 P 2dee97c110914202b2e9ca8db83df0bf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:13:39.285573 26776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:39.285275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40693 (local address 127.9.155.65:48339)
0620 19:13:39.285423 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:39.285490 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 19:13:39.449862 26777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:39.449419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40693 (local address 127.9.155.66:42627)
0620 19:13:39.449681 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:39.449744 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":72,"threads_started":1}
I20260620 19:13:39.653017 26778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:39.652525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40693 (local address 127.9.155.67:42485)
0620 19:13:39.652842 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:39.652908 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":68,"threads_started":1}
I20260620 19:13:40.286906 26779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:40.286449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40693 (local address 127.9.155.65:56049)
0620 19:13:40.286741 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:40.286804 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":77,"threads_started":1}
I20260620 19:13:40.451072 26780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:40.450624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40693 (local address 127.9.155.66:37843)
0620 19:13:40.450900 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:40.450963 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":79,"threads_started":1}
I20260620 19:13:40.654418 26781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:40.653976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40693 (local address 127.9.155.67:54837)
0620 19:13:40.654234 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:40.654290 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":81,"threads_started":1}
I20260620 19:13:41.233174 26782 raft_consensus.cc:482] T 463e435ef88445068b9a83b60d90c288 P 2dee97c110914202b2e9ca8db83df0bf [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:13:41.233330 26782 raft_consensus.cc:504] T 463e435ef88445068b9a83b60d90c288 P 2dee97c110914202b2e9ca8db83df0bf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a830c89ba25b4f8cb3b47d173e20ff3e" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 45875 } } peers { permanent_uuid: "0b46980251d54a8bb398e0da234a85ec" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 38157 } } peers { permanent_uuid: "2dee97c110914202b2e9ca8db83df0bf" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 39169 } }
I20260620 19:13:41.233719 26782 leader_election.cc:288] T 463e435ef88445068b9a83b60d90c288 P 2dee97c110914202b2e9ca8db83df0bf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a830c89ba25b4f8cb3b47d173e20ff3e (127.9.155.66:45875), 0b46980251d54a8bb398e0da234a85ec (127.9.155.67:38157)
W20260620 19:13:41.234489 26387 leader_election.cc:341] T 463e435ef88445068b9a83b60d90c288 P 2dee97c110914202b2e9ca8db83df0bf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a830c89ba25b4f8cb3b47d173e20ff3e (127.9.155.66:45875): Not found: Tablet not found: 463e435ef88445068b9a83b60d90c288
W20260620 19:13:41.234587 26386 leader_election.cc:341] T 463e435ef88445068b9a83b60d90c288 P 2dee97c110914202b2e9ca8db83df0bf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0b46980251d54a8bb398e0da234a85ec (127.9.155.67:38157): Not found: Tablet not found: 463e435ef88445068b9a83b60d90c288
I20260620 19:13:41.234671 26386 leader_election.cc:302] T 463e435ef88445068b9a83b60d90c288 P 2dee97c110914202b2e9ca8db83df0bf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2dee97c110914202b2e9ca8db83df0bf; no voters: 0b46980251d54a8bb398e0da234a85ec, a830c89ba25b4f8cb3b47d173e20ff3e
I20260620 19:13:41.234799 26782 raft_consensus.cc:2726] T 463e435ef88445068b9a83b60d90c288 P 2dee97c110914202b2e9ca8db83df0bf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:13:41.288058 26783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:41.287648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40693 (local address 127.9.155.65:55727)
0620 19:13:41.287899 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:41.287952 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":87,"threads_started":1}
I20260620 19:13:41.452208 26784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:41.451816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40693 (local address 127.9.155.66:60901)
0620 19:13:41.452059 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:41.452114 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":87,"threads_started":1}
I20260620 19:13:41.655654 26785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:41.655217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40693 (local address 127.9.155.67:37787)
0620 19:13:41.655496 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:41.655560 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":80,"threads_started":1}
I20260620 19:13:42.289458 26786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:42.288987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40693 (local address 127.9.155.65:34733)
0620 19:13:42.289278 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:42.289361 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":75,"threads_started":1}
I20260620 19:13:42.453481 26787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:42.453033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40693 (local address 127.9.155.66:38031)
0620 19:13:42.453296 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:42.453360 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":90,"threads_started":1}
I20260620 19:13:42.656977 26788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:42.656474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40693 (local address 127.9.155.67:52053)
0620 19:13:42.656763 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:42.656871 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":79,"threads_started":1}
I20260620 19:13:43.290694 26789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:43.290208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40693 (local address 127.9.155.65:39923)
0620 19:13:43.290535 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:43.290597 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":118,"threads_started":1}
I20260620 19:13:43.454721 26790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:43.454234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40693 (local address 127.9.155.66:47627)
0620 19:13:43.454543 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:43.454620 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":101,"threads_started":1}
I20260620 19:13:43.658337 26791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:43.657816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40693 (local address 127.9.155.67:35439)
0620 19:13:43.658128 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:43.658201 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":97,"threads_started":1}
I20260620 19:13:44.292007 26792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:44.291597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40693 (local address 127.9.155.65:58557)
0620 19:13:44.291852 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:44.291913 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":76,"threads_started":1}
I20260620 19:13:44.455875 26793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:44.455431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40693 (local address 127.9.155.66:47469)
0620 19:13:44.455721 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:44.455779 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":70,"threads_started":1}
I20260620 19:13:44.471199 26794 raft_consensus.cc:482] T 463e435ef88445068b9a83b60d90c288 P 2dee97c110914202b2e9ca8db83df0bf [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:13:44.471333 26794 raft_consensus.cc:504] T 463e435ef88445068b9a83b60d90c288 P 2dee97c110914202b2e9ca8db83df0bf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a830c89ba25b4f8cb3b47d173e20ff3e" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 45875 } } peers { permanent_uuid: "0b46980251d54a8bb398e0da234a85ec" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 38157 } } peers { permanent_uuid: "2dee97c110914202b2e9ca8db83df0bf" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 39169 } }
I20260620 19:13:44.471763 26794 leader_election.cc:288] T 463e435ef88445068b9a83b60d90c288 P 2dee97c110914202b2e9ca8db83df0bf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a830c89ba25b4f8cb3b47d173e20ff3e (127.9.155.66:45875), 0b46980251d54a8bb398e0da234a85ec (127.9.155.67:38157)
W20260620 19:13:44.472616 26387 leader_election.cc:341] T 463e435ef88445068b9a83b60d90c288 P 2dee97c110914202b2e9ca8db83df0bf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a830c89ba25b4f8cb3b47d173e20ff3e (127.9.155.66:45875): Not found: Tablet not found: 463e435ef88445068b9a83b60d90c288
W20260620 19:13:44.472723 26386 leader_election.cc:341] T 463e435ef88445068b9a83b60d90c288 P 2dee97c110914202b2e9ca8db83df0bf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0b46980251d54a8bb398e0da234a85ec (127.9.155.67:38157): Not found: Tablet not found: 463e435ef88445068b9a83b60d90c288
I20260620 19:13:44.472818 26386 leader_election.cc:302] T 463e435ef88445068b9a83b60d90c288 P 2dee97c110914202b2e9ca8db83df0bf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2dee97c110914202b2e9ca8db83df0bf; no voters: 0b46980251d54a8bb398e0da234a85ec, a830c89ba25b4f8cb3b47d173e20ff3e
I20260620 19:13:44.472999 26794 raft_consensus.cc:2726] T 463e435ef88445068b9a83b60d90c288 P 2dee97c110914202b2e9ca8db83df0bf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:13:44.659539 26795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:44.659067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40693 (local address 127.9.155.67:52985)
0620 19:13:44.659385 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:44.659440 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":81,"threads_started":1}
I20260620 19:13:45.293380 26796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:45.292903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40693 (local address 127.9.155.65:52975)
0620 19:13:45.293201 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:45.293268 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":77,"threads_started":1}
I20260620 19:13:45.457178 26797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:45.456691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40693 (local address 127.9.155.66:47295)
0620 19:13:45.457015 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:45.457079 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":91,"threads_started":1}
I20260620 19:13:45.660907 26798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:45.660454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40693 (local address 127.9.155.67:53139)
0620 19:13:45.660730 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:45.660797 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":88,"threads_started":1}
I20260620 19:13:46.294677 26799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:46.294246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40693 (local address 127.9.155.65:38049)
0620 19:13:46.294521 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:46.294582 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":91,"threads_started":1}
I20260620 19:13:46.458460 26800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:46.458020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40693 (local address 127.9.155.66:48043)
0620 19:13:46.458300 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:46.458362 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":96,"threads_started":1}
I20260620 19:13:46.662250 26801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:46.661760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40693 (local address 127.9.155.67:42417)
0620 19:13:46.662076 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:46.662135 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":94,"threads_started":1}
I20260620 19:13:47.296085 26802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:47.295641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40693 (local address 127.9.155.65:42129)
0620 19:13:47.295915 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:47.295987 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":81,"threads_started":1}
I20260620 19:13:47.459970 26803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:47.459470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40693 (local address 127.9.155.66:49103)
0620 19:13:47.459754 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:47.459867 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":78,"threads_started":1}
I20260620 19:13:47.663511 26804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:47.663093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40693 (local address 127.9.155.67:44149)
0620 19:13:47.663353 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:47.663415 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":71,"threads_started":1}
I20260620 19:13:48.297358 26805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:48.296906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40693 (local address 127.9.155.65:43055)
0620 19:13:48.297192 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:48.297255 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":116,"threads_started":1}
I20260620 19:13:48.461202 26806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:48.460765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40693 (local address 127.9.155.66:53349)
0620 19:13:48.461044 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:48.461106 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":88,"threads_started":1}
I20260620 19:13:48.664721 26807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:48.664280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40693 (local address 127.9.155.67:37351)
0620 19:13:48.664560 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:48.664623 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":85,"threads_started":1}
I20260620 19:13:49.013849 26808 raft_consensus.cc:482] T 463e435ef88445068b9a83b60d90c288 P 2dee97c110914202b2e9ca8db83df0bf [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:13:49.014034 26808 raft_consensus.cc:504] T 463e435ef88445068b9a83b60d90c288 P 2dee97c110914202b2e9ca8db83df0bf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a830c89ba25b4f8cb3b47d173e20ff3e" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 45875 } } peers { permanent_uuid: "0b46980251d54a8bb398e0da234a85ec" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 38157 } } peers { permanent_uuid: "2dee97c110914202b2e9ca8db83df0bf" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 39169 } }
I20260620 19:13:49.014463 26808 leader_election.cc:288] T 463e435ef88445068b9a83b60d90c288 P 2dee97c110914202b2e9ca8db83df0bf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a830c89ba25b4f8cb3b47d173e20ff3e (127.9.155.66:45875), 0b46980251d54a8bb398e0da234a85ec (127.9.155.67:38157)
W20260620 19:13:49.015232 26386 leader_election.cc:341] T 463e435ef88445068b9a83b60d90c288 P 2dee97c110914202b2e9ca8db83df0bf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0b46980251d54a8bb398e0da234a85ec (127.9.155.67:38157): Not found: Tablet not found: 463e435ef88445068b9a83b60d90c288
W20260620 19:13:49.015343 26387 leader_election.cc:341] T 463e435ef88445068b9a83b60d90c288 P 2dee97c110914202b2e9ca8db83df0bf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a830c89ba25b4f8cb3b47d173e20ff3e (127.9.155.66:45875): Not found: Tablet not found: 463e435ef88445068b9a83b60d90c288
I20260620 19:13:49.015430 26387 leader_election.cc:302] T 463e435ef88445068b9a83b60d90c288 P 2dee97c110914202b2e9ca8db83df0bf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2dee97c110914202b2e9ca8db83df0bf; no voters: 0b46980251d54a8bb398e0da234a85ec, a830c89ba25b4f8cb3b47d173e20ff3e
I20260620 19:13:49.015569 26808 raft_consensus.cc:2726] T 463e435ef88445068b9a83b60d90c288 P 2dee97c110914202b2e9ca8db83df0bf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:13:49.298715 26809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:49.298230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40693 (local address 127.9.155.65:35315)
0620 19:13:49.298543 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:49.298614 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":92,"threads_started":1}
I20260620 19:13:49.462415 26810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:49.461960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40693 (local address 127.9.155.66:41985)
0620 19:13:49.462240 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:49.462303 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":77,"threads_started":1}
I20260620 19:13:49.666044 26811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:49.665601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40693 (local address 127.9.155.67:53185)
0620 19:13:49.665855 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:49.665917 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":77,"threads_started":1}
I20260620 19:13:50.300010 26812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:50.299474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40693 (local address 127.9.155.65:33499)
0620 19:13:50.299846 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:50.299913 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":98,"threads_started":1}
I20260620 19:13:50.463721 26813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:50.463254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40693 (local address 127.9.155.66:58567)
0620 19:13:50.463549 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:50.463619 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":77,"threads_started":1}
I20260620 19:13:50.667927 26814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:50.667508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40693 (local address 127.9.155.67:43721)
0620 19:13:50.667761 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:50.667819 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":74,"threads_started":1}
I20260620 19:13:51.301383 26815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:51.300910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40693 (local address 127.9.155.65:40189)
0620 19:13:51.301214 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:51.301281 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":84,"threads_started":1}
I20260620 19:13:51.464967 26816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:51.464554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40693 (local address 127.9.155.66:33695)
0620 19:13:51.464808 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:51.464869 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":74,"threads_started":1}
I20260620 19:13:51.669406 26817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:51.668755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40693 (local address 127.9.155.67:42369)
0620 19:13:51.669205 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:51.669280 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":79,"threads_started":1}
I20260620 19:13:52.302991 26818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:52.302513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40693 (local address 127.9.155.65:55663)
0620 19:13:52.302773 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:52.302886 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":84,"threads_started":1}
I20260620 19:13:52.466269 26819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:52.465836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:40693 (local address 127.9.155.66:40515)
0620 19:13:52.466108 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:52.466170 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.126:40693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":76,"threads_started":1}
I20260620 19:13:52.648044 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 26329
I20260620 19:13:52.653445 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 26502
I20260620 19:13:52.658617 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 26636
2026-06-20T19:13:52Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59526 ms)
[----------] 30 tests from SecurityITest (237419 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:13:52 dist-test-slave-n3fx krb5kdc[26822](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:13:52 dist-test-slave-n3fx krb5kdc[26822](info): set up 2 sockets
Jun 20 19:13:52 dist-test-slave-n3fx krb5kdc[26822](info): commencing operation
krb5kdc: starting...
W20260620 19:13:53.623891 9837 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/krb5kdc/test-user.keytab.
Jun 20 19:13:53 dist-test-slave-n3fx krb5kdc[26822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:13:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:13:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.155.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.155.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.155.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab.
Entry for principal kudu/127.9.155.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab.
I20260620 19:13:53.739189 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:42721
--webserver_interface=127.9.155.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab
--principal=kudu/127.9.155.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:35841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.155.126:42721
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:13:53.851467 26835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:13:53.851783 26835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:13:53.851840 26835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:13:53.855968 26835 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:13:53.856048 26835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:13:53.856077 26835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:13:53.856097 26835 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:13:53.856117 26835 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:13:53.861660 26835 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:35841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.155.126:42721
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab
--principal=kudu/127.9.155.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:42721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.155.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:13:53.862754 26835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:13:53.863757 26835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:13:53.872551 26840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:13:53.872638 26835 server_base.cc:1034] running on GCE node
W20260620 19:13:53.872551 26841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:13:53.872555 26843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:13:53.873296 26835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:13:53.874450 26835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:13:53.875623 26835 hybrid_clock.cc:648] HybridClock initialized: now 1781982833875603 us; error 36 us; skew 500 ppm
Jun 20 19:13:53 dist-test-slave-n3fx krb5kdc[26822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982833, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:13:53.878548 26835 init.cc:377] Logged in from keytab as kudu/127.9.155.126@KRBTEST.COM (short username kudu)
I20260620 19:13:53.879487 26835 webserver.cc:460] Webserver started at http://127.9.155.126:35821/ using document root <none> and password file <none>
I20260620 19:13:53.880031 26835 fs_manager.cc:359] Metadata directory not provided
I20260620 19:13:53.880097 26835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:13:53.880358 26835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:13:53.882404 26835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/minicluster-data/master-0/data/instance:
uuid: "b69bf0813b6c4d7287efad5471129428"
format_stamp: "Formatted at 2026-06-20 19:13:53 on dist-test-slave-n3fx"
server_key: "58142b25d9630f6f3d97e721dcb8c8a8"
server_key_iv: "4addf177d8626670e2acdaac94179ccb"
server_key_version: "encryptionkey@0"
I20260620 19:13:53.882936 26835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/minicluster-data/master-0/wal/instance:
uuid: "b69bf0813b6c4d7287efad5471129428"
format_stamp: "Formatted at 2026-06-20 19:13:53 on dist-test-slave-n3fx"
server_key: "58142b25d9630f6f3d97e721dcb8c8a8"
server_key_iv: "4addf177d8626670e2acdaac94179ccb"
server_key_version: "encryptionkey@0"
I20260620 19:13:53.886699 26835 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 19:13:53.889238 26850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:53.890589 26835 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:13:53.890710 26835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/minicluster-data/master-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/minicluster-data/master-0/wal
uuid: "b69bf0813b6c4d7287efad5471129428"
format_stamp: "Formatted at 2026-06-20 19:13:53 on dist-test-slave-n3fx"
server_key: "58142b25d9630f6f3d97e721dcb8c8a8"
server_key_iv: "4addf177d8626670e2acdaac94179ccb"
server_key_version: "encryptionkey@0"
I20260620 19:13:53.890828 26835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:13:53.914295 26835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:13:53.917495 26835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:13:53.917675 26835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:13:53.924870 26835 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.126:42721
I20260620 19:13:53.924886 26902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.126:42721 every 8 connection(s)
I20260620 19:13:53.925877 26835 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
I20260620 19:13:53.928947 26903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:13:53.934836 26903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b69bf0813b6c4d7287efad5471129428: Bootstrap starting.
I20260620 19:13:53.935820 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 26835
I20260620 19:13:53.935981 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/minicluster-data/master-0/wal/instance
I20260620 19:13:53.936287 9837 external_mini_cluster.cc:1428] Setting key 723e010ff349254517bdcd0bf692e282
I20260620 19:13:53.937749 26903 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b69bf0813b6c4d7287efad5471129428: Neither blocks nor log segments found. Creating new log.
I20260620 19:13:53.938501 26903 log.cc:824] T 00000000000000000000000000000000 P b69bf0813b6c4d7287efad5471129428: Log is configured to *not* fsync() on all Append() calls
I20260620 19:13:53.940771 26903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b69bf0813b6c4d7287efad5471129428: No bootstrap required, opened a new log
I20260620 19:13:53.945075 26903 raft_consensus.cc:348] T 00000000000000000000000000000000 P b69bf0813b6c4d7287efad5471129428 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b69bf0813b6c4d7287efad5471129428" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 42721 } }
Jun 20 19:13:53 dist-test-slave-n3fx krb5kdc[26822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:13:53.945319 26903 raft_consensus.cc:374] T 00000000000000000000000000000000 P b69bf0813b6c4d7287efad5471129428 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:13:53.945407 26903 raft_consensus.cc:729] T 00000000000000000000000000000000 P b69bf0813b6c4d7287efad5471129428 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b69bf0813b6c4d7287efad5471129428, State: Initialized, Role: FOLLOWER
I20260620 19:13:53.945887 26903 consensus_queue.cc:260] T 00000000000000000000000000000000 P b69bf0813b6c4d7287efad5471129428 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b69bf0813b6c4d7287efad5471129428" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 42721 } }
I20260620 19:13:53.946019 26903 raft_consensus.cc:388] T 00000000000000000000000000000000 P b69bf0813b6c4d7287efad5471129428 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:13:53.946079 26903 raft_consensus.cc:482] T 00000000000000000000000000000000 P b69bf0813b6c4d7287efad5471129428 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:13:53.946173 26903 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b69bf0813b6c4d7287efad5471129428 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:13:53.947074 26903 raft_consensus.cc:504] T 00000000000000000000000000000000 P b69bf0813b6c4d7287efad5471129428 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b69bf0813b6c4d7287efad5471129428" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 42721 } }
I20260620 19:13:53.947403 26903 leader_election.cc:302] T 00000000000000000000000000000000 P b69bf0813b6c4d7287efad5471129428 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b69bf0813b6c4d7287efad5471129428; no voters:
I20260620 19:13:53.947700 26903 leader_election.cc:288] T 00000000000000000000000000000000 P b69bf0813b6c4d7287efad5471129428 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:13:53.947791 26908 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b69bf0813b6c4d7287efad5471129428 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:13:53.947959 26908 raft_consensus.cc:686] T 00000000000000000000000000000000 P b69bf0813b6c4d7287efad5471129428 [term 1 LEADER]: Becoming Leader. State: Replica: b69bf0813b6c4d7287efad5471129428, State: Running, Role: LEADER
I20260620 19:13:53.948359 26908 consensus_queue.cc:237] T 00000000000000000000000000000000 P b69bf0813b6c4d7287efad5471129428 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b69bf0813b6c4d7287efad5471129428" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 42721 } }
I20260620 19:13:53.949069 26903 sys_catalog.cc:564] T 00000000000000000000000000000000 P b69bf0813b6c4d7287efad5471129428 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:13:53.949575 26908 sys_catalog.cc:455] T 00000000000000000000000000000000 P b69bf0813b6c4d7287efad5471129428 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b69bf0813b6c4d7287efad5471129428. Latest consensus state: current_term: 1 leader_uuid: "b69bf0813b6c4d7287efad5471129428" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b69bf0813b6c4d7287efad5471129428" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 42721 } } }
I20260620 19:13:53.949786 26908 sys_catalog.cc:458] T 00000000000000000000000000000000 P b69bf0813b6c4d7287efad5471129428 [sys.catalog]: This master's current role is: LEADER
I20260620 19:13:53.950209 26915 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:13:53.950719 26906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:53.937432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39150 (local address 127.9.155.126:42721)
0620 19:13:53.937895 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:53.937911 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:53.937945 (+ 34us) server_negotiation.cc:407] Connection header received
0620 19:13:53.938605 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:53.938624 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:53.939025 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:53.939388 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:53.940305 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:53.941383 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:53.942547 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:53.943044 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:53.943163 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:53.945848 (+ 2685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:13:53.945855 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:13:53.945874 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:13:53.945908 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:13:53.947576 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:53.948079 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:53.948085 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:53.948091 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:53.948248 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:53.948517 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:53.948521 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:53.948523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:53.948934 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:13:53.949142 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:53.949461 (+ 319us) server_negotiation.cc:299] Negotiation successful
0620 19:13:53.949771 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":144,"threads_started":1}
I20260620 19:13:53.951438 26909 sys_catalog.cc:455] T 00000000000000000000000000000000 P b69bf0813b6c4d7287efad5471129428 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b69bf0813b6c4d7287efad5471129428" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b69bf0813b6c4d7287efad5471129428" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 42721 } } }
I20260620 19:13:53.952708 26909 sys_catalog.cc:458] T 00000000000000000000000000000000 P b69bf0813b6c4d7287efad5471129428 [sys.catalog]: This master's current role is: LEADER
I20260620 19:13:53.953634 26915 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:13:53.959749 26915 catalog_manager.cc:1347] Generated new cluster ID: a03099f6bdf1456dae31aec2424a38c1
I20260620 19:13:53.959836 26915 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:13:53.980564 26915 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:13:53.981577 26915 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:13:53.985929 26915 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b69bf0813b6c4d7287efad5471129428: Generated new TSK 0
I20260620 19:13:53.986793 26915 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.155.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.155.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.155.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.65.keytab.
Entry for principal kudu/127.9.155.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.65.keytab.
I20260620 19:13:54.025494 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.65:0
--local_ip_for_outbound_sockets=127.9.155.65
--webserver_interface=127.9.155.65
--webserver_port=0
--tserver_master_addrs=127.9.155.126:42721
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.65.keytab
--principal=kudu/127.9.155.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:35841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:13:54.156349 26929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:13:54.156605 26929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:13:54.156646 26929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:13:54.160671 26929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:13:54.160737 26929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:13:54.160797 26929 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.65
I20260620 19:13:54.166501 26929 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:35841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.65.keytab
--principal=kudu/127.9.155.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.155.65
--webserver_port=0
--tserver_master_addrs=127.9.155.126:42721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26929
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.65
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:13:54.167672 26929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:13:54.168769 26929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:13:54.178277 26934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:13:54.178375 26935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:13:54.178437 26929 server_base.cc:1034] running on GCE node
W20260620 19:13:54.178557 26937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:13:54.178977 26929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:13:54.179664 26929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:13:54.180845 26929 hybrid_clock.cc:648] HybridClock initialized: now 1781982834180823 us; error 34 us; skew 500 ppm
Jun 20 19:13:54 dist-test-slave-n3fx krb5kdc[26822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982834, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:13:54.183660 26929 init.cc:377] Logged in from keytab as kudu/127.9.155.65@KRBTEST.COM (short username kudu)
I20260620 19:13:54.184677 26929 webserver.cc:460] Webserver started at http://127.9.155.65:45519/ using document root <none> and password file <none>
I20260620 19:13:54.185214 26929 fs_manager.cc:359] Metadata directory not provided
I20260620 19:13:54.185276 26929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:13:54.185469 26929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:13:54.187530 26929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/minicluster-data/ts-0/data/instance:
uuid: "a1e79b0058934696a12d2062e3aeb525"
format_stamp: "Formatted at 2026-06-20 19:13:54 on dist-test-slave-n3fx"
server_key: "13a4f6b491468edb626b376753800126"
server_key_iv: "134a77edb3ca6ddf8f6b33c50723dbf0"
server_key_version: "encryptionkey@0"
I20260620 19:13:54.188081 26929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/minicluster-data/ts-0/wal/instance:
uuid: "a1e79b0058934696a12d2062e3aeb525"
format_stamp: "Formatted at 2026-06-20 19:13:54 on dist-test-slave-n3fx"
server_key: "13a4f6b491468edb626b376753800126"
server_key_iv: "134a77edb3ca6ddf8f6b33c50723dbf0"
server_key_version: "encryptionkey@0"
I20260620 19:13:54.191841 26929 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 19:13:54.194357 26944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:54.195658 26929 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 19:13:54.195756 26929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/minicluster-data/ts-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/minicluster-data/ts-0/wal
uuid: "a1e79b0058934696a12d2062e3aeb525"
format_stamp: "Formatted at 2026-06-20 19:13:54 on dist-test-slave-n3fx"
server_key: "13a4f6b491468edb626b376753800126"
server_key_iv: "134a77edb3ca6ddf8f6b33c50723dbf0"
server_key_version: "encryptionkey@0"
I20260620 19:13:54.195858 26929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:13:54.217901 26929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:13:54.221236 26929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:13:54.221465 26929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:13:54.222100 26929 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:13:54.222975 26929 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:13:54.223023 26929 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:54.223064 26929 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:13:54.223093 26929 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:54.231839 26929 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.65:42777
I20260620 19:13:54.231866 27057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.65:42777 every 8 connection(s)
I20260620 19:13:54.232776 26929 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
Jun 20 19:13:54 dist-test-slave-n3fx krb5kdc[26822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982834, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.65@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:13:54.242901 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 26929
I20260620 19:13:54.243016 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/minicluster-data/ts-0/wal/instance
I20260620 19:13:54.243312 9837 external_mini_cluster.cc:1428] Setting key 398edc9ebb6ca4f148411d4d79aa2b0c
I20260620 19:13:54.246670 26906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:54.234627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.65:48135 (local address 127.9.155.126:42721)
0620 19:13:54.234771 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:54.234776 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:54.235656 (+ 880us) server_negotiation.cc:407] Connection header received
0620 19:13:54.236591 (+ 935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:54.236593 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:54.236643 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:54.236724 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:54.237974 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:54.238545 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:54.239755 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:54.240016 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:54.240110 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:54.242660 (+ 2550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:13:54.242663 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:13:54.242665 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:13:54.242694 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:13:54.244384 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:54.244986 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:54.244990 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:54.244992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:54.245098 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:54.245422 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:54.245425 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:54.245427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:54.245664 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:13:54.245766 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:54.246379 (+ 613us) server_negotiation.cc:299] Negotiation successful
0620 19:13:54.246530 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 19:13:54.247824 27060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:54.235002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:42721 (local address 127.9.155.65:48135)
0620 19:13:54.235487 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:54.235533 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:54.236375 (+ 842us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:54.236925 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:54.236934 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:54.237290 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:54.237797 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:54.237813 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:54.238675 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:54.238678 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:54.239618 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:54.239629 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:54.240171 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:54.240174 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:54.240362 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:54.240955 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:54.240972 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:54.242479 (+ 1507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:13:54.244502 (+ 2023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:54.244509 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:54.244529 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:54.244884 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:54.245191 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:54.245193 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:54.245194 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:54.245336 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:54.245797 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:13:54.245804 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:13:54.246132 (+ 328us) client_negotiation.cc:769] Sending connection context
0620 19:13:54.246316 (+ 184us) client_negotiation.cc:240] Negotiation successful
0620 19:13:54.246711 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":116,"threads_started":1}
I20260620 19:13:54.249109 27058 heartbeater.cc:347] Connected to a master server at 127.9.155.126:42721
I20260620 19:13:54.249349 27058 heartbeater.cc:464] Registering TS with master...
I20260620 19:13:54.249946 27058 heartbeater.cc:511] Master 127.9.155.126:42721 requested a full tablet report, sending...
I20260620 19:13:54.251639 26867 ts_manager.cc:194] Registered new tserver with Master: a1e79b0058934696a12d2062e3aeb525 (127.9.155.65:42777)
WARNING: no policy specified for kudu/127.9.155.66@KRBTEST.COM; defaulting to no policy
I20260620 19:13:54.253265 26867 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.155.65@KRBTEST.COM'} at 127.9.155.65:48135
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.155.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.155.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.66.keytab.
Entry for principal kudu/127.9.155.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.66.keytab.
I20260620 19:13:54.268767 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.66:0
--local_ip_for_outbound_sockets=127.9.155.66
--webserver_interface=127.9.155.66
--webserver_port=0
--tserver_master_addrs=127.9.155.126:42721
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.66.keytab
--principal=kudu/127.9.155.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:35841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:13:54.393342 27063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:13:54.393620 27063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:13:54.393682 27063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:13:54.397933 27063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:13:54.398021 27063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:13:54.398105 27063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.66
I20260620 19:13:54.403599 27063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:35841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.66.keytab
--principal=kudu/127.9.155.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.155.66
--webserver_port=0
--tserver_master_addrs=127.9.155.126:42721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.66
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:13:54.404757 27063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:13:54.405748 27063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:13:54.415311 27071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:13:54.415663 27068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:13:54.415779 27063 server_base.cc:1034] running on GCE node
W20260620 19:13:54.415828 27069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:13:54.416365 27063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:13:54.417050 27063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:13:54.418228 27063 hybrid_clock.cc:648] HybridClock initialized: now 1781982834418194 us; error 50 us; skew 500 ppm
Jun 20 19:13:54 dist-test-slave-n3fx krb5kdc[26822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982834, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:13:54.421155 27063 init.cc:377] Logged in from keytab as kudu/127.9.155.66@KRBTEST.COM (short username kudu)
I20260620 19:13:54.422140 27063 webserver.cc:460] Webserver started at http://127.9.155.66:40535/ using document root <none> and password file <none>
I20260620 19:13:54.422690 27063 fs_manager.cc:359] Metadata directory not provided
I20260620 19:13:54.422755 27063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:13:54.422932 27063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:13:54.425064 27063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/minicluster-data/ts-1/data/instance:
uuid: "50caa040513545029d0fd35a67cc1a5d"
format_stamp: "Formatted at 2026-06-20 19:13:54 on dist-test-slave-n3fx"
server_key: "9315cc23d626d253c6fd7ce3920f51e7"
server_key_iv: "348ac979b3934426b672c2ec68d61935"
server_key_version: "encryptionkey@0"
I20260620 19:13:54.425602 27063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/minicluster-data/ts-1/wal/instance:
uuid: "50caa040513545029d0fd35a67cc1a5d"
format_stamp: "Formatted at 2026-06-20 19:13:54 on dist-test-slave-n3fx"
server_key: "9315cc23d626d253c6fd7ce3920f51e7"
server_key_iv: "348ac979b3934426b672c2ec68d61935"
server_key_version: "encryptionkey@0"
I20260620 19:13:54.429524 27063 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 19:13:54.432080 27078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:54.433331 27063 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 19:13:54.433444 27063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/minicluster-data/ts-1/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/minicluster-data/ts-1/wal
uuid: "50caa040513545029d0fd35a67cc1a5d"
format_stamp: "Formatted at 2026-06-20 19:13:54 on dist-test-slave-n3fx"
server_key: "9315cc23d626d253c6fd7ce3920f51e7"
server_key_iv: "348ac979b3934426b672c2ec68d61935"
server_key_version: "encryptionkey@0"
I20260620 19:13:54.433558 27063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:13:54.452544 27063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:13:54.455747 27063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:13:54.455919 27063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:13:54.456532 27063 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:13:54.457348 27063 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:13:54.457396 27063 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:54.457450 27063 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:13:54.457485 27063 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:54.465652 27063 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.66:37601
I20260620 19:13:54.465708 27191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.66:37601 every 8 connection(s)
I20260620 19:13:54.466528 27063 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
I20260620 19:13:54.475469 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 27063
Jun 20 19:13:54 dist-test-slave-n3fx krb5kdc[26822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982834, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.66@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:13:54.475589 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/minicluster-data/ts-1/wal/instance
I20260620 19:13:54.475857 9837 external_mini_cluster.cc:1428] Setting key b93fe609fc0cf879ecd756c9b8257bcd
I20260620 19:13:54.479669 26906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:54.468396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.66:45693 (local address 127.9.155.126:42721)
0620 19:13:54.468551 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:54.468555 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:54.469438 (+ 883us) server_negotiation.cc:407] Connection header received
0620 19:13:54.470377 (+ 939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:54.470380 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:54.470427 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:54.470504 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:54.471677 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:54.472255 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:54.473419 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:54.473653 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:54.473736 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:54.476117 (+ 2381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:13:54.476120 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:13:54.476122 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:13:54.476146 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:13:54.477543 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:54.478031 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:54.478035 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:54.478036 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:54.478115 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:54.478434 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:54.478437 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:54.478439 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:54.478688 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:13:54.478787 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:54.479342 (+ 555us) server_negotiation.cc:299] Negotiation successful
0620 19:13:54.479501 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 19:13:54.480763 27194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:54.468752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:42721 (local address 127.9.155.66:45693)
0620 19:13:54.469260 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:54.469317 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:54.470160 (+ 843us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:54.470645 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:54.470651 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:54.471039 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:54.471530 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:54.471545 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:54.472378 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:54.472382 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:54.473301 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:54.473310 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:54.473785 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:54.473788 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:54.473980 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:54.474525 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:54.474541 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:54.475951 (+ 1410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:13:54.477649 (+ 1698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:54.477655 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:54.477670 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:54.477935 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:54.478192 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:54.478196 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:54.478198 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:54.478340 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:54.478816 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:13:54.478821 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:13:54.479100 (+ 279us) client_negotiation.cc:769] Sending connection context
0620 19:13:54.479282 (+ 182us) client_negotiation.cc:240] Negotiation successful
0620 19:13:54.479663 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":126,"threads_started":1}
I20260620 19:13:54.482100 27192 heartbeater.cc:347] Connected to a master server at 127.9.155.126:42721
I20260620 19:13:54.482338 27192 heartbeater.cc:464] Registering TS with master...
I20260620 19:13:54.482877 27192 heartbeater.cc:511] Master 127.9.155.126:42721 requested a full tablet report, sending...
I20260620 19:13:54.484017 26867 ts_manager.cc:194] Registered new tserver with Master: 50caa040513545029d0fd35a67cc1a5d (127.9.155.66:37601)
I20260620 19:13:54.484683 26867 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.155.66@KRBTEST.COM'} at 127.9.155.66:45693
WARNING: no policy specified for kudu/127.9.155.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.155.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.155.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.67.keytab.
Entry for principal kudu/127.9.155.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.67.keytab.
I20260620 19:13:54.504607 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.67:0
--local_ip_for_outbound_sockets=127.9.155.67
--webserver_interface=127.9.155.67
--webserver_port=0
--tserver_master_addrs=127.9.155.126:42721
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.67.keytab
--principal=kudu/127.9.155.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:35841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:13:54.614202 27197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:13:54.614456 27197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:13:54.614500 27197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:13:54.618557 27197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:13:54.618631 27197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:13:54.618700 27197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.67
I20260620 19:13:54.624203 27197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:35841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.67.keytab
--principal=kudu/127.9.155.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.155.67
--webserver_port=0
--tserver_master_addrs=127.9.155.126:42721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.67
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:13:54.625308 27197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:13:54.626371 27197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:13:54.636191 27202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:13:54.636184 27205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:13:54.636184 27203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:13:54.636294 27197 server_base.cc:1034] running on GCE node
I20260620 19:13:54.636983 27197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:13:54.637665 27197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:13:54.638826 27197 hybrid_clock.cc:648] HybridClock initialized: now 1781982834638801 us; error 42 us; skew 500 ppm
Jun 20 19:13:54 dist-test-slave-n3fx krb5kdc[26822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982834, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:13:54.641672 27197 init.cc:377] Logged in from keytab as kudu/127.9.155.67@KRBTEST.COM (short username kudu)
I20260620 19:13:54.642534 27197 webserver.cc:460] Webserver started at http://127.9.155.67:44605/ using document root <none> and password file <none>
I20260620 19:13:54.643065 27197 fs_manager.cc:359] Metadata directory not provided
I20260620 19:13:54.643131 27197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:13:54.643332 27197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:13:54.645399 27197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/minicluster-data/ts-2/data/instance:
uuid: "16d04b16278a4655a00843abddf9d920"
format_stamp: "Formatted at 2026-06-20 19:13:54 on dist-test-slave-n3fx"
server_key: "7752cf143d2156432b3e4994e5c9da55"
server_key_iv: "d8855183a9361f91c2a83970cd599cc2"
server_key_version: "encryptionkey@0"
I20260620 19:13:54.645957 27197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/minicluster-data/ts-2/wal/instance:
uuid: "16d04b16278a4655a00843abddf9d920"
format_stamp: "Formatted at 2026-06-20 19:13:54 on dist-test-slave-n3fx"
server_key: "7752cf143d2156432b3e4994e5c9da55"
server_key_iv: "d8855183a9361f91c2a83970cd599cc2"
server_key_version: "encryptionkey@0"
I20260620 19:13:54.650013 27197 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 19:13:54.652920 27212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:54.654425 27197 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:13:54.654551 27197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/minicluster-data/ts-2/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/minicluster-data/ts-2/wal
uuid: "16d04b16278a4655a00843abddf9d920"
format_stamp: "Formatted at 2026-06-20 19:13:54 on dist-test-slave-n3fx"
server_key: "7752cf143d2156432b3e4994e5c9da55"
server_key_iv: "d8855183a9361f91c2a83970cd599cc2"
server_key_version: "encryptionkey@0"
I20260620 19:13:54.654682 27197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:13:54.687801 27197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:13:54.691107 27197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:13:54.691289 27197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:13:54.691938 27197 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:13:54.692909 27197 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:13:54.692958 27197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:54.693004 27197 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:13:54.693032 27197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:54.701754 27197 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.67:45669
I20260620 19:13:54.701813 27325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.67:45669 every 8 connection(s)
I20260620 19:13:54.702675 27197 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
I20260620 19:13:54.711458 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 27197
I20260620 19:13:54.711613 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/minicluster-data/ts-2/wal/instance
I20260620 19:13:54.712009 9837 external_mini_cluster.cc:1428] Setting key 5d78e53e170b7c69011463becfe3f07f
Jun 20 19:13:54 dist-test-slave-n3fx krb5kdc[26822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982834, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.67@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:13:54.716506 26906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:54.704537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.67:50297 (local address 127.9.155.126:42721)
0620 19:13:54.704669 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:54.704672 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:54.705537 (+ 865us) server_negotiation.cc:407] Connection header received
0620 19:13:54.706484 (+ 947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:54.706487 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:54.706548 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:54.706626 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:54.707859 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:54.708446 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:54.709616 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:54.709842 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:54.709934 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:54.712799 (+ 2865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:13:54.712803 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:13:54.712805 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:13:54.712834 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:13:54.714507 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:54.715011 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:54.715013 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:54.715015 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:54.715084 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:54.715396 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:54.715399 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:54.715400 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:54.715602 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:13:54.715683 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:54.716250 (+ 567us) server_negotiation.cc:299] Negotiation successful
0620 19:13:54.716364 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 19:13:54.717307 27328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:54.704881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:42721 (local address 127.9.155.67:50297)
0620 19:13:54.705376 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:54.705420 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:54.706264 (+ 844us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:54.706812 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:54.706819 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:54.707207 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:54.707705 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:54.707719 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:54.708581 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:54.708584 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:54.709493 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:54.709503 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:54.709988 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:54.709992 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:54.710165 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:54.710756 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:54.710773 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:54.712597 (+ 1824us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:13:54.714610 (+ 2013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:54.714615 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:54.714627 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:54.714883 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:54.715149 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:54.715151 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:54.715152 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:54.715280 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:54.715699 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:13:54.715705 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:13:54.715992 (+ 287us) client_negotiation.cc:769] Sending connection context
0620 19:13:54.716151 (+ 159us) client_negotiation.cc:240] Negotiation successful
0620 19:13:54.716472 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":112,"threads_started":1}
I20260620 19:13:54.718457 27326 heartbeater.cc:347] Connected to a master server at 127.9.155.126:42721
I20260620 19:13:54.718680 27326 heartbeater.cc:464] Registering TS with master...
I20260620 19:13:54.719162 27326 heartbeater.cc:511] Master 127.9.155.126:42721 requested a full tablet report, sending...
I20260620 19:13:54.720260 26867 ts_manager.cc:194] Registered new tserver with Master: 16d04b16278a4655a00843abddf9d920 (127.9.155.67:45669)
I20260620 19:13:54.720837 26867 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.155.67@KRBTEST.COM'} at 127.9.155.67:50297
I20260620 19:13:54.726068 9837 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:13:54.735363 26906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:54.728079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39154 (local address 127.9.155.126:42721)
0620 19:13:54.728250 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:54.728254 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:54.728331 (+ 77us) server_negotiation.cc:407] Connection header received
0620 19:13:54.728440 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:54.728442 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:54.728485 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:54.728552 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:54.729172 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:54.729692 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:54.730574 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:54.730792 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:54.730878 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:54.732254 (+ 1376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:13:54.732259 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:13:54.732261 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:13:54.732290 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:13:54.733578 (+ 1288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:54.734046 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:54.734050 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:54.734051 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:54.734144 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:54.734468 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:54.734471 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:54.734472 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:54.734704 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:13:54.734806 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:54.735124 (+ 318us) server_negotiation.cc:299] Negotiation successful
0620 19:13:54.735225 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 19:13:54.738420 26867 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39154:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:13:54.741567 26867 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:13:54.755852 27340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:54.749778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.67:45669 (local address 127.0.0.1:41972)
0620 19:13:54.751108 (+ 1330us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:54.751121 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:54.751222 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:54.751669 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:54.751674 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:54.751694 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:13:54.752080 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:54.752083 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:54.753065 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:54.753069 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:54.754306 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:54.754316 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:54.755264 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:54.755268 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:54.755392 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:54.755439 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 19:13:54.755556 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 19:13:54.755700 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1240,"mutex_wait_us":272,"spinlock_wait_cycles":2304,"thread_start_us":78,"threads_started":1}
I20260620 19:13:54.755870 27339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:54.749958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41972 (local address 127.9.155.67:45669)
0620 19:13:54.750438 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:54.750450 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:54.751228 (+ 778us) server_negotiation.cc:407] Connection header received
0620 19:13:54.751322 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:54.751328 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:54.751535 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:54.751669 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:13:54.752165 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:54.752945 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:54.754429 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:54.755143 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:54.755246 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:54.755395 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:54.755595 (+ 200us) server_negotiation.cc:299] Negotiation successful
0620 19:13:54.755699 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":140,"threads_started":1}
I20260620 19:13:54.756198 27338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:54.749718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.66:37601 (local address 127.0.0.1:47542)
0620 19:13:54.750497 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:54.750541 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:54.750654 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:54.751712 (+ 1058us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:54.751715 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:54.751744 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:13:54.751935 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:54.751942 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:54.753374 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:54.753378 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:54.754788 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:54.754798 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:54.755832 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:54.755836 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:54.755932 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:54.755956 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 19:13:54.756005 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 19:13:54.756073 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"mutex_wait_us":132,"spinlock_wait_cycles":896,"thread_start_us":81,"threads_started":1}
I20260620 19:13:54.756755 27337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:54.750499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.65:42777 (local address 127.0.0.1:59508)
0620 19:13:54.751291 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:54.751303 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:54.751417 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:54.751900 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:54.751903 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:54.751918 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:13:54.751992 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:54.751997 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:54.753374 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:54.753378 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:54.754776 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:54.754785 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:54.755859 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:54.755862 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:54.755959 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:54.755981 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 19:13:54.756087 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 19:13:54.756162 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":731,"mutex_wait_us":146,"thread_start_us":77,"threads_started":1}
I20260620 19:13:54.756754 27341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:54.750709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59508 (local address 127.9.155.65:42777)
0620 19:13:54.751009 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:54.751015 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:54.751376 (+ 361us) server_negotiation.cc:407] Connection header received
0620 19:13:54.751513 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:54.751517 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:54.751671 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:54.751789 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:13:54.752094 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:54.753241 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:54.754904 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:54.755573 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:54.755666 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:54.755799 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:54.756131 (+ 332us) server_negotiation.cc:299] Negotiation successful
0620 19:13:54.756263 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":118,"threads_started":1}
I20260620 19:13:54.759002 27260 tablet_service.cc:1467] Processing CreateTablet for tablet 18632bdc8b8b49b0b5f9e61e067b2b75 (DEFAULT_TABLE table=test-table [id=c4c40447cec34f37afb42dae1e2a5ad1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:13:54.760057 27260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18632bdc8b8b49b0b5f9e61e067b2b75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:13:54.759591 26992 tablet_service.cc:1467] Processing CreateTablet for tablet 18632bdc8b8b49b0b5f9e61e067b2b75 (DEFAULT_TABLE table=test-table [id=c4c40447cec34f37afb42dae1e2a5ad1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:13:54.760623 26992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18632bdc8b8b49b0b5f9e61e067b2b75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:13:54.764408 27342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:54.750897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47542 (local address 127.9.155.66:37601)
0620 19:13:54.751270 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:54.751282 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:54.751303 (+ 21us) server_negotiation.cc:407] Connection header received
0620 19:13:54.751371 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:54.751376 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:54.751561 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:54.751849 (+ 288us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:13:54.752014 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:54.753241 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:54.754924 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:54.755702 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:54.756499 (+ 797us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:54.756638 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:54.756685 (+ 47us) server_negotiation.cc:299] Negotiation successful
0620 19:13:54.756771 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":146,"threads_started":1}
I20260620 19:13:54.767057 27344 tablet_bootstrap.cc:492] T 18632bdc8b8b49b0b5f9e61e067b2b75 P 16d04b16278a4655a00843abddf9d920: Bootstrap starting.
I20260620 19:13:54.767771 27126 tablet_service.cc:1467] Processing CreateTablet for tablet 18632bdc8b8b49b0b5f9e61e067b2b75 (DEFAULT_TABLE table=test-table [id=c4c40447cec34f37afb42dae1e2a5ad1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:13:54.768599 27343 tablet_bootstrap.cc:492] T 18632bdc8b8b49b0b5f9e61e067b2b75 P a1e79b0058934696a12d2062e3aeb525: Bootstrap starting.
I20260620 19:13:54.768786 27126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18632bdc8b8b49b0b5f9e61e067b2b75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:13:54.769621 27344 tablet_bootstrap.cc:654] T 18632bdc8b8b49b0b5f9e61e067b2b75 P 16d04b16278a4655a00843abddf9d920: Neither blocks nor log segments found. Creating new log.
I20260620 19:13:54.770889 27343 tablet_bootstrap.cc:654] T 18632bdc8b8b49b0b5f9e61e067b2b75 P a1e79b0058934696a12d2062e3aeb525: Neither blocks nor log segments found. Creating new log.
I20260620 19:13:54.770891 27344 log.cc:824] T 18632bdc8b8b49b0b5f9e61e067b2b75 P 16d04b16278a4655a00843abddf9d920: Log is configured to *not* fsync() on all Append() calls
I20260620 19:13:54.771730 27343 log.cc:824] T 18632bdc8b8b49b0b5f9e61e067b2b75 P a1e79b0058934696a12d2062e3aeb525: Log is configured to *not* fsync() on all Append() calls
I20260620 19:13:54.773280 27344 tablet_bootstrap.cc:492] T 18632bdc8b8b49b0b5f9e61e067b2b75 P 16d04b16278a4655a00843abddf9d920: No bootstrap required, opened a new log
I20260620 19:13:54.773515 27344 ts_tablet_manager.cc:1397] T 18632bdc8b8b49b0b5f9e61e067b2b75 P 16d04b16278a4655a00843abddf9d920: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260620 19:13:54.773602 27343 tablet_bootstrap.cc:492] T 18632bdc8b8b49b0b5f9e61e067b2b75 P a1e79b0058934696a12d2062e3aeb525: No bootstrap required, opened a new log
I20260620 19:13:54.773777 27343 ts_tablet_manager.cc:1397] T 18632bdc8b8b49b0b5f9e61e067b2b75 P a1e79b0058934696a12d2062e3aeb525: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 19:13:54.776469 27347 tablet_bootstrap.cc:492] T 18632bdc8b8b49b0b5f9e61e067b2b75 P 50caa040513545029d0fd35a67cc1a5d: Bootstrap starting.
I20260620 19:13:54.777462 27343 raft_consensus.cc:348] T 18632bdc8b8b49b0b5f9e61e067b2b75 P a1e79b0058934696a12d2062e3aeb525 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50caa040513545029d0fd35a67cc1a5d" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 37601 } } peers { permanent_uuid: "16d04b16278a4655a00843abddf9d920" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 45669 } } peers { permanent_uuid: "a1e79b0058934696a12d2062e3aeb525" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 42777 } }
I20260620 19:13:54.777422 27344 raft_consensus.cc:348] T 18632bdc8b8b49b0b5f9e61e067b2b75 P 16d04b16278a4655a00843abddf9d920 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50caa040513545029d0fd35a67cc1a5d" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 37601 } } peers { permanent_uuid: "16d04b16278a4655a00843abddf9d920" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 45669 } } peers { permanent_uuid: "a1e79b0058934696a12d2062e3aeb525" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 42777 } }
I20260620 19:13:54.777726 27344 raft_consensus.cc:374] T 18632bdc8b8b49b0b5f9e61e067b2b75 P 16d04b16278a4655a00843abddf9d920 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:13:54.777729 27343 raft_consensus.cc:374] T 18632bdc8b8b49b0b5f9e61e067b2b75 P a1e79b0058934696a12d2062e3aeb525 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:13:54.777784 27343 raft_consensus.cc:729] T 18632bdc8b8b49b0b5f9e61e067b2b75 P a1e79b0058934696a12d2062e3aeb525 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1e79b0058934696a12d2062e3aeb525, State: Initialized, Role: FOLLOWER
I20260620 19:13:54.777784 27344 raft_consensus.cc:729] T 18632bdc8b8b49b0b5f9e61e067b2b75 P 16d04b16278a4655a00843abddf9d920 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16d04b16278a4655a00843abddf9d920, State: Initialized, Role: FOLLOWER
I20260620 19:13:54.778245 27343 consensus_queue.cc:260] T 18632bdc8b8b49b0b5f9e61e067b2b75 P a1e79b0058934696a12d2062e3aeb525 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50caa040513545029d0fd35a67cc1a5d" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 37601 } } peers { permanent_uuid: "16d04b16278a4655a00843abddf9d920" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 45669 } } peers { permanent_uuid: "a1e79b0058934696a12d2062e3aeb525" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 42777 } }
I20260620 19:13:54.778257 27344 consensus_queue.cc:260] T 18632bdc8b8b49b0b5f9e61e067b2b75 P 16d04b16278a4655a00843abddf9d920 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50caa040513545029d0fd35a67cc1a5d" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 37601 } } peers { permanent_uuid: "16d04b16278a4655a00843abddf9d920" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 45669 } } peers { permanent_uuid: "a1e79b0058934696a12d2062e3aeb525" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 42777 } }
I20260620 19:13:54.778882 27347 tablet_bootstrap.cc:654] T 18632bdc8b8b49b0b5f9e61e067b2b75 P 50caa040513545029d0fd35a67cc1a5d: Neither blocks nor log segments found. Creating new log.
I20260620 19:13:54.779206 27058 heartbeater.cc:503] Master 127.9.155.126:42721 was elected leader, sending a full tablet report...
I20260620 19:13:54.779213 27326 heartbeater.cc:503] Master 127.9.155.126:42721 was elected leader, sending a full tablet report...
I20260620 19:13:54.779331 27344 ts_tablet_manager.cc:1428] T 18632bdc8b8b49b0b5f9e61e067b2b75 P 16d04b16278a4655a00843abddf9d920: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 19:13:54.779415 27343 ts_tablet_manager.cc:1428] T 18632bdc8b8b49b0b5f9e61e067b2b75 P a1e79b0058934696a12d2062e3aeb525: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 19:13:54.781124 27347 log.cc:824] T 18632bdc8b8b49b0b5f9e61e067b2b75 P 50caa040513545029d0fd35a67cc1a5d: Log is configured to *not* fsync() on all Append() calls
I20260620 19:13:54.783185 27347 tablet_bootstrap.cc:492] T 18632bdc8b8b49b0b5f9e61e067b2b75 P 50caa040513545029d0fd35a67cc1a5d: No bootstrap required, opened a new log
I20260620 19:13:54.783390 27347 ts_tablet_manager.cc:1397] T 18632bdc8b8b49b0b5f9e61e067b2b75 P 50caa040513545029d0fd35a67cc1a5d: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20260620 19:13:54.786619 27347 raft_consensus.cc:348] T 18632bdc8b8b49b0b5f9e61e067b2b75 P 50caa040513545029d0fd35a67cc1a5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50caa040513545029d0fd35a67cc1a5d" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 37601 } } peers { permanent_uuid: "16d04b16278a4655a00843abddf9d920" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 45669 } } peers { permanent_uuid: "a1e79b0058934696a12d2062e3aeb525" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 42777 } }
I20260620 19:13:54.786895 27347 raft_consensus.cc:374] T 18632bdc8b8b49b0b5f9e61e067b2b75 P 50caa040513545029d0fd35a67cc1a5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:13:54.786939 27347 raft_consensus.cc:729] T 18632bdc8b8b49b0b5f9e61e067b2b75 P 50caa040513545029d0fd35a67cc1a5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50caa040513545029d0fd35a67cc1a5d, State: Initialized, Role: FOLLOWER
I20260620 19:13:54.787364 27347 consensus_queue.cc:260] T 18632bdc8b8b49b0b5f9e61e067b2b75 P 50caa040513545029d0fd35a67cc1a5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50caa040513545029d0fd35a67cc1a5d" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 37601 } } peers { permanent_uuid: "16d04b16278a4655a00843abddf9d920" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 45669 } } peers { permanent_uuid: "a1e79b0058934696a12d2062e3aeb525" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 42777 } }
I20260620 19:13:54.788054 27192 heartbeater.cc:503] Master 127.9.155.126:42721 was elected leader, sending a full tablet report...
I20260620 19:13:54.788341 27347 ts_tablet_manager.cc:1428] T 18632bdc8b8b49b0b5f9e61e067b2b75 P 50caa040513545029d0fd35a67cc1a5d: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260620 19:13:54.832803 27349 raft_consensus.cc:482] T 18632bdc8b8b49b0b5f9e61e067b2b75 P a1e79b0058934696a12d2062e3aeb525 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:13:54.833022 27349 raft_consensus.cc:504] T 18632bdc8b8b49b0b5f9e61e067b2b75 P a1e79b0058934696a12d2062e3aeb525 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50caa040513545029d0fd35a67cc1a5d" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 37601 } } peers { permanent_uuid: "16d04b16278a4655a00843abddf9d920" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 45669 } } peers { permanent_uuid: "a1e79b0058934696a12d2062e3aeb525" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 42777 } }
I20260620 19:13:54.834203 27349 leader_election.cc:288] T 18632bdc8b8b49b0b5f9e61e067b2b75 P a1e79b0058934696a12d2062e3aeb525 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 50caa040513545029d0fd35a67cc1a5d (127.9.155.66:37601), 16d04b16278a4655a00843abddf9d920 (127.9.155.67:45669)
I20260620 19:13:54.838209 27339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:54.834572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.65:41229 (local address 127.9.155.67:45669)
0620 19:13:54.834749 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:54.834754 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:54.834891 (+ 137us) server_negotiation.cc:407] Connection header received
0620 19:13:54.835022 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:54.835025 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:54.835072 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:54.835138 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:13:54.835455 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:54.836001 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:54.837165 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:54.837596 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:54.837702 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:54.837757 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:54.838010 (+ 253us) server_negotiation.cc:299] Negotiation successful
0620 19:13:54.838087 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260620 19:13:54.838207 27342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:54.834514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.65:33439 (local address 127.9.155.66:37601)
0620 19:13:54.834633 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:54.834646 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:54.834877 (+ 231us) server_negotiation.cc:407] Connection header received
0620 19:13:54.835098 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:54.835101 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:54.835152 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:54.835227 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:13:54.835499 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:54.835977 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:54.837130 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:54.837590 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:54.837701 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:54.837771 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:54.837997 (+ 226us) server_negotiation.cc:299] Negotiation successful
0620 19:13:54.838069 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 19:13:54.838546 27352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:54.834414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.66:37601 (local address 127.9.155.65:33439)
0620 19:13:54.834779 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:54.834793 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:54.834984 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:54.835247 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:54.835250 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:54.835265 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:13:54.835391 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:54.835397 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:54.836078 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:54.836081 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:54.837024 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:54.837034 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:54.837702 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:54.837705 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:54.837889 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:54.837911 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 19:13:54.837982 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 19:13:54.838121 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"mutex_wait_us":110,"thread_start_us":127,"threads_started":1}
I20260620 19:13:54.838570 27353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:54.834497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.67:45669 (local address 127.9.155.65:41229)
0620 19:13:54.834797 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:54.834810 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:54.834919 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:54.835162 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:54.835165 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:54.835180 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:13:54.835355 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:54.835363 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:54.836110 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:54.836113 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:54.837054 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:54.837063 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:54.837702 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:54.837705 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:54.837872 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:54.837906 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 19:13:54.837982 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 19:13:54.838122 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"mutex_wait_us":79,"thread_start_us":111,"threads_started":1}
I20260620 19:13:54.839222 27280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18632bdc8b8b49b0b5f9e61e067b2b75" candidate_uuid: "a1e79b0058934696a12d2062e3aeb525" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16d04b16278a4655a00843abddf9d920" is_pre_election: true
I20260620 19:13:54.839360 27146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18632bdc8b8b49b0b5f9e61e067b2b75" candidate_uuid: "a1e79b0058934696a12d2062e3aeb525" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50caa040513545029d0fd35a67cc1a5d" is_pre_election: true
I20260620 19:13:54.839548 27280 raft_consensus.cc:2454] T 18632bdc8b8b49b0b5f9e61e067b2b75 P 16d04b16278a4655a00843abddf9d920 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1e79b0058934696a12d2062e3aeb525 in term 0.
I20260620 19:13:54.839650 27146 raft_consensus.cc:2454] T 18632bdc8b8b49b0b5f9e61e067b2b75 P 50caa040513545029d0fd35a67cc1a5d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1e79b0058934696a12d2062e3aeb525 in term 0.
I20260620 19:13:54.840006 26947 leader_election.cc:302] T 18632bdc8b8b49b0b5f9e61e067b2b75 P a1e79b0058934696a12d2062e3aeb525 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16d04b16278a4655a00843abddf9d920, a1e79b0058934696a12d2062e3aeb525; no voters:
I20260620 19:13:54.840292 27349 raft_consensus.cc:2781] T 18632bdc8b8b49b0b5f9e61e067b2b75 P a1e79b0058934696a12d2062e3aeb525 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:13:54.840382 27349 raft_consensus.cc:482] T 18632bdc8b8b49b0b5f9e61e067b2b75 P a1e79b0058934696a12d2062e3aeb525 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:13:54.840440 27349 raft_consensus.cc:3037] T 18632bdc8b8b49b0b5f9e61e067b2b75 P a1e79b0058934696a12d2062e3aeb525 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:13:54.841455 27349 raft_consensus.cc:504] T 18632bdc8b8b49b0b5f9e61e067b2b75 P a1e79b0058934696a12d2062e3aeb525 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50caa040513545029d0fd35a67cc1a5d" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 37601 } } peers { permanent_uuid: "16d04b16278a4655a00843abddf9d920" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 45669 } } peers { permanent_uuid: "a1e79b0058934696a12d2062e3aeb525" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 42777 } }
I20260620 19:13:54.841779 27349 leader_election.cc:288] T 18632bdc8b8b49b0b5f9e61e067b2b75 P a1e79b0058934696a12d2062e3aeb525 [CANDIDATE]: Term 1 election: Requested vote from peers 50caa040513545029d0fd35a67cc1a5d (127.9.155.66:37601), 16d04b16278a4655a00843abddf9d920 (127.9.155.67:45669)
I20260620 19:13:54.842175 27146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18632bdc8b8b49b0b5f9e61e067b2b75" candidate_uuid: "a1e79b0058934696a12d2062e3aeb525" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50caa040513545029d0fd35a67cc1a5d"
I20260620 19:13:54.842288 27146 raft_consensus.cc:3037] T 18632bdc8b8b49b0b5f9e61e067b2b75 P 50caa040513545029d0fd35a67cc1a5d [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:13:54.843345 27146 raft_consensus.cc:2454] T 18632bdc8b8b49b0b5f9e61e067b2b75 P 50caa040513545029d0fd35a67cc1a5d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1e79b0058934696a12d2062e3aeb525 in term 1.
I20260620 19:13:54.843637 27280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18632bdc8b8b49b0b5f9e61e067b2b75" candidate_uuid: "a1e79b0058934696a12d2062e3aeb525" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16d04b16278a4655a00843abddf9d920"
I20260620 19:13:54.843727 26948 leader_election.cc:302] T 18632bdc8b8b49b0b5f9e61e067b2b75 P a1e79b0058934696a12d2062e3aeb525 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50caa040513545029d0fd35a67cc1a5d, a1e79b0058934696a12d2062e3aeb525; no voters:
I20260620 19:13:54.843760 27280 raft_consensus.cc:3037] T 18632bdc8b8b49b0b5f9e61e067b2b75 P 16d04b16278a4655a00843abddf9d920 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:13:54.843954 27349 raft_consensus.cc:2781] T 18632bdc8b8b49b0b5f9e61e067b2b75 P a1e79b0058934696a12d2062e3aeb525 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:13:54.844206 27349 raft_consensus.cc:686] T 18632bdc8b8b49b0b5f9e61e067b2b75 P a1e79b0058934696a12d2062e3aeb525 [term 1 LEADER]: Becoming Leader. State: Replica: a1e79b0058934696a12d2062e3aeb525, State: Running, Role: LEADER
I20260620 19:13:54.844614 27349 consensus_queue.cc:237] T 18632bdc8b8b49b0b5f9e61e067b2b75 P a1e79b0058934696a12d2062e3aeb525 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50caa040513545029d0fd35a67cc1a5d" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 37601 } } peers { permanent_uuid: "16d04b16278a4655a00843abddf9d920" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 45669 } } peers { permanent_uuid: "a1e79b0058934696a12d2062e3aeb525" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 42777 } }
I20260620 19:13:54.845062 27280 raft_consensus.cc:2454] T 18632bdc8b8b49b0b5f9e61e067b2b75 P 16d04b16278a4655a00843abddf9d920 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1e79b0058934696a12d2062e3aeb525 in term 1.
I20260620 19:13:54.847882 26867 catalog_manager.cc:5495] T 18632bdc8b8b49b0b5f9e61e067b2b75 P a1e79b0058934696a12d2062e3aeb525 reported cstate change: term changed from 0 to 1, leader changed from <none> to a1e79b0058934696a12d2062e3aeb525 (127.9.155.65). New cstate: current_term: 1 leader_uuid: "a1e79b0058934696a12d2062e3aeb525" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50caa040513545029d0fd35a67cc1a5d" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 37601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16d04b16278a4655a00843abddf9d920" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 45669 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1e79b0058934696a12d2062e3aeb525" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 42777 } health_report { overall_health: HEALTHY } } }
I20260620 19:13:54.860750 27341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:54.857167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59520 (local address 127.9.155.65:42777)
0620 19:13:54.857274 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:54.857277 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:54.857290 (+ 13us) server_negotiation.cc:407] Connection header received
0620 19:13:54.857364 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:54.857367 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:54.857421 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:54.857509 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:13:54.857690 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:54.858301 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:54.859231 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:54.859464 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:54.859561 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:54.859856 (+ 295us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:13:54.860350 (+ 494us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:13:54.860464 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:54.860545 (+ 81us) server_negotiation.cc:299] Negotiation successful
0620 19:13:54.860610 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 19:13:54.866757 27146 raft_consensus.cc:1261] T 18632bdc8b8b49b0b5f9e61e067b2b75 P 50caa040513545029d0fd35a67cc1a5d [term 1 FOLLOWER]: Refusing update from remote peer a1e79b0058934696a12d2062e3aeb525: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:13:54.866757 27280 raft_consensus.cc:1261] T 18632bdc8b8b49b0b5f9e61e067b2b75 P 16d04b16278a4655a00843abddf9d920 [term 1 FOLLOWER]: Refusing update from remote peer a1e79b0058934696a12d2062e3aeb525: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:13:54.867483 27349 consensus_queue.cc:1035] T 18632bdc8b8b49b0b5f9e61e067b2b75 P a1e79b0058934696a12d2062e3aeb525 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d04b16278a4655a00843abddf9d920" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 45669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:13:54.867676 27354 consensus_queue.cc:1035] T 18632bdc8b8b49b0b5f9e61e067b2b75 P a1e79b0058934696a12d2062e3aeb525 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50caa040513545029d0fd35a67cc1a5d" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 37601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:13:54.875960 27359 mvcc.cc:205] Tried to move back new op lower bound from 7299001691607019520 to 7299001691526946816. Current Snapshot: MvccSnapshot[applied={T|T < 7299001691607019520}]
I20260620 19:13:54.886629 26866 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:13:54.886790 26866 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:13:54.889281 26866 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b69bf0813b6c4d7287efad5471129428: Sending DeleteTablet for 3 replicas of tablet 18632bdc8b8b49b0b5f9e61e067b2b75
I20260620 19:13:54.890092 27126 tablet_service.cc:1514] Processing DeleteTablet for tablet 18632bdc8b8b49b0b5f9e61e067b2b75 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:13:54 UTC) from {username='kudu', principal='kudu/127.9.155.126@KRBTEST.COM'} at 127.0.0.1:47542
I20260620 19:13:54.890223 26992 tablet_service.cc:1514] Processing DeleteTablet for tablet 18632bdc8b8b49b0b5f9e61e067b2b75 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:13:54 UTC) from {username='kudu', principal='kudu/127.9.155.126@KRBTEST.COM'} at 127.0.0.1:59508
I20260620 19:13:54.890228 27260 tablet_service.cc:1514] Processing DeleteTablet for tablet 18632bdc8b8b49b0b5f9e61e067b2b75 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:13:54 UTC) from {username='kudu', principal='kudu/127.9.155.126@KRBTEST.COM'} at 127.0.0.1:41972
I20260620 19:13:54.890702 27370 tablet_replica.cc:330] T 18632bdc8b8b49b0b5f9e61e067b2b75 P 16d04b16278a4655a00843abddf9d920: stopping tablet replica
I20260620 19:13:54.891003 27370 raft_consensus.cc:2229] T 18632bdc8b8b49b0b5f9e61e067b2b75 P 16d04b16278a4655a00843abddf9d920 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:13:54.891144 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 26929
I20260620 19:13:54.892161 27370 raft_consensus.cc:2258] T 18632bdc8b8b49b0b5f9e61e067b2b75 P 16d04b16278a4655a00843abddf9d920 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:13:54.892578 27369 tablet_replica.cc:330] T 18632bdc8b8b49b0b5f9e61e067b2b75 P 50caa040513545029d0fd35a67cc1a5d: stopping tablet replica
I20260620 19:13:54.892884 27369 raft_consensus.cc:2229] T 18632bdc8b8b49b0b5f9e61e067b2b75 P 50caa040513545029d0fd35a67cc1a5d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:13:54.893149 27369 raft_consensus.cc:2258] T 18632bdc8b8b49b0b5f9e61e067b2b75 P 50caa040513545029d0fd35a67cc1a5d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:13:54.893791 27370 ts_tablet_manager.cc:1905] T 18632bdc8b8b49b0b5f9e61e067b2b75 P 16d04b16278a4655a00843abddf9d920: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:13:54.894560 27369 ts_tablet_manager.cc:1905] T 18632bdc8b8b49b0b5f9e61e067b2b75 P 50caa040513545029d0fd35a67cc1a5d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:13:54.896194 27370 ts_tablet_manager.cc:1918] T 18632bdc8b8b49b0b5f9e61e067b2b75 P 16d04b16278a4655a00843abddf9d920: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:13:54.896314 27370 log.cc:1196] T 18632bdc8b8b49b0b5f9e61e067b2b75 P 16d04b16278a4655a00843abddf9d920: Deleting WAL directory at /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/minicluster-data/ts-2/wal/wals/18632bdc8b8b49b0b5f9e61e067b2b75
I20260620 19:13:54.896605 27370 ts_tablet_manager.cc:1939] T 18632bdc8b8b49b0b5f9e61e067b2b75 P 16d04b16278a4655a00843abddf9d920: Deleting consensus metadata
I20260620 19:13:54.897017 27369 ts_tablet_manager.cc:1918] T 18632bdc8b8b49b0b5f9e61e067b2b75 P 50caa040513545029d0fd35a67cc1a5d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:13:54.897111 27369 log.cc:1196] T 18632bdc8b8b49b0b5f9e61e067b2b75 P 50caa040513545029d0fd35a67cc1a5d: Deleting WAL directory at /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982595231230-9837-0/minicluster-data/ts-1/wal/wals/18632bdc8b8b49b0b5f9e61e067b2b75
I20260620 19:13:54.897384 27369 ts_tablet_manager.cc:1939] T 18632bdc8b8b49b0b5f9e61e067b2b75 P 50caa040513545029d0fd35a67cc1a5d: Deleting consensus metadata
I20260620 19:13:54.897768 26853 catalog_manager.cc:4841] TS 16d04b16278a4655a00843abddf9d920 (127.9.155.67:45669): tablet 18632bdc8b8b49b0b5f9e61e067b2b75 (table test-table [id=c4c40447cec34f37afb42dae1e2a5ad1]) successfully deleted
W20260620 19:13:54.898494 26853 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.155.65:42777 (error 108)
I20260620 19:13:54.898680 26854 catalog_manager.cc:4841] TS 50caa040513545029d0fd35a67cc1a5d (127.9.155.66:37601): tablet 18632bdc8b8b49b0b5f9e61e067b2b75 (table test-table [id=c4c40447cec34f37afb42dae1e2a5ad1]) successfully deleted
I20260620 19:13:54.898712 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 27063
I20260620 19:13:54.901070 27337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:54.900773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.65:42777 (local address 127.0.0.1:59522)
0620 19:13:54.900903 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:54.900973 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.65:42777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 19:13:54.901239 26853 catalog_manager.cc:4584] TS a1e79b0058934696a12d2062e3aeb525 (127.9.155.65:42777): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 18632bdc8b8b49b0b5f9e61e067b2b75: Network error: Client connection negotiation failed: client connection to 127.9.155.65:42777: connect: Connection refused (error 111)
I20260620 19:13:54.903468 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 27197
I20260620 19:13:54.908008 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 26835
2026-06-20T19:13:54Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2249 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-20T19:13:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:13:54Z Disabled control of system clock
I20260620 19:13:54.936023 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982595231230-9837-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982595231230-9837-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982595231230-9837-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:46025
--webserver_interface=127.9.155.126
--webserver_port=0
--builtin_ntp_servers=127.9.155.84:37773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.155.126:46025
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 19:13:55.045751 27376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:13:55.046056 27376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:13:55.046096 27376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:13:55.050217 27376 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:13:55.050292 27376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:13:55.050307 27376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:13:55.050324 27376 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:13:55.050338 27376 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:13:55.055270 27376 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:37773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982595231230-9837-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982595231230-9837-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.155.126:46025
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:46025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.155.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27376
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982595231230-9837-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:13:55.056401 27376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:13:55.057758 27376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 19:13:55.066615 27376 server_base.cc:1034] running on GCE node
W20260620 19:13:55.066568 27384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:13:55.066622 27382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:13:55.066792 27381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:13:55.067415 27376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:13:55.068573 27376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:13:55.069737 27376 hybrid_clock.cc:648] HybridClock initialized: now 1781982835069685 us; error 68 us; skew 500 ppm
I20260620 19:13:55.071702 27376 webserver.cc:460] Webserver started at http://127.9.155.126:33049/ using document root <none> and password file <none>
I20260620 19:13:55.072333 27376 fs_manager.cc:359] Metadata directory not provided
I20260620 19:13:55.072418 27376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:13:55.072625 27376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:13:55.074671 27376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982595231230-9837-0/minicluster-data/master-0/data/instance:
uuid: "ead6b34e9a5f408487edfe2e29e91eec"
format_stamp: "Formatted at 2026-06-20 19:13:55 on dist-test-slave-n3fx"
server_key: "80263a8f9594f14a3bfccbfef1c434f8"
server_key_iv: "eaddaa7e49457f69c78e806dbeeb0817"
server_key_version: "encryptionkey@0"
I20260620 19:13:55.075201 27376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982595231230-9837-0/minicluster-data/master-0/wal/instance:
uuid: "ead6b34e9a5f408487edfe2e29e91eec"
format_stamp: "Formatted at 2026-06-20 19:13:55 on dist-test-slave-n3fx"
server_key: "80263a8f9594f14a3bfccbfef1c434f8"
server_key_iv: "eaddaa7e49457f69c78e806dbeeb0817"
server_key_version: "encryptionkey@0"
I20260620 19:13:55.078959 27376 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 19:13:55.081538 27390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:55.082932 27376 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 19:13:55.083073 27376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982595231230-9837-0/minicluster-data/master-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982595231230-9837-0/minicluster-data/master-0/wal
uuid: "ead6b34e9a5f408487edfe2e29e91eec"
format_stamp: "Formatted at 2026-06-20 19:13:55 on dist-test-slave-n3fx"
server_key: "80263a8f9594f14a3bfccbfef1c434f8"
server_key_iv: "eaddaa7e49457f69c78e806dbeeb0817"
server_key_version: "encryptionkey@0"
I20260620 19:13:55.083184 27376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982595231230-9837-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982595231230-9837-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982595231230-9837-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:13:55.096721 27376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:13:55.097404 27376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:13:55.097568 27376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:13:55.104310 27376 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.126:46025
I20260620 19:13:55.104357 27442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.126:46025 every 8 connection(s)
I20260620 19:13:55.105268 27376 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
I20260620 19:13:55.108461 27443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:13:55.111871 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 27376
I20260620 19:13:55.111994 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982595231230-9837-0/minicluster-data/master-0/wal/instance
I20260620 19:13:55.112344 9837 external_mini_cluster.cc:1428] Setting key aa0c10a5bfbedb6011d6e1d4dbee1ed2
I20260620 19:13:55.114777 27443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ead6b34e9a5f408487edfe2e29e91eec: Bootstrap starting.
I20260620 19:13:55.117141 27443 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ead6b34e9a5f408487edfe2e29e91eec: Neither blocks nor log segments found. Creating new log.
I20260620 19:13:55.117898 27443 log.cc:824] T 00000000000000000000000000000000 P ead6b34e9a5f408487edfe2e29e91eec: Log is configured to *not* fsync() on all Append() calls
I20260620 19:13:55.119756 27443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ead6b34e9a5f408487edfe2e29e91eec: No bootstrap required, opened a new log
I20260620 19:13:55.121743 27446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:55.113989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49964 (local address 127.9.155.126:46025)
0620 19:13:55.114705 (+ 716us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:55.114721 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:55.114758 (+ 37us) server_negotiation.cc:407] Connection header received
0620 19:13:55.115507 (+ 749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:55.115527 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:55.115951 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:55.116306 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:55.116767 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:55.117770 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:55.118664 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:55.119130 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:55.119272 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:55.120058 (+ 786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:13:55.120063 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:13:55.120075 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:13:55.120512 (+ 437us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:13:55.120569 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:13:55.120590 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:13:55.120674 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:55.120798 (+ 124us) server_negotiation.cc:299] Negotiation successful
0620 19:13:55.121050 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":151,"threads_started":1}
I20260620 19:13:55.123874 27443 raft_consensus.cc:348] T 00000000000000000000000000000000 P ead6b34e9a5f408487edfe2e29e91eec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ead6b34e9a5f408487edfe2e29e91eec" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 46025 } }
I20260620 19:13:55.124167 27443 raft_consensus.cc:374] T 00000000000000000000000000000000 P ead6b34e9a5f408487edfe2e29e91eec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:13:55.124246 27443 raft_consensus.cc:729] T 00000000000000000000000000000000 P ead6b34e9a5f408487edfe2e29e91eec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ead6b34e9a5f408487edfe2e29e91eec, State: Initialized, Role: FOLLOWER
I20260620 19:13:55.124634 27443 consensus_queue.cc:260] T 00000000000000000000000000000000 P ead6b34e9a5f408487edfe2e29e91eec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ead6b34e9a5f408487edfe2e29e91eec" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 46025 } }
I20260620 19:13:55.124751 27443 raft_consensus.cc:388] T 00000000000000000000000000000000 P ead6b34e9a5f408487edfe2e29e91eec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:13:55.124788 27443 raft_consensus.cc:482] T 00000000000000000000000000000000 P ead6b34e9a5f408487edfe2e29e91eec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:13:55.124855 27443 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ead6b34e9a5f408487edfe2e29e91eec [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:13:55.125748 27443 raft_consensus.cc:504] T 00000000000000000000000000000000 P ead6b34e9a5f408487edfe2e29e91eec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ead6b34e9a5f408487edfe2e29e91eec" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 46025 } }
I20260620 19:13:55.126046 27443 leader_election.cc:302] T 00000000000000000000000000000000 P ead6b34e9a5f408487edfe2e29e91eec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ead6b34e9a5f408487edfe2e29e91eec; no voters:
I20260620 19:13:55.126299 27443 leader_election.cc:288] T 00000000000000000000000000000000 P ead6b34e9a5f408487edfe2e29e91eec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:13:55.126422 27448 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ead6b34e9a5f408487edfe2e29e91eec [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:13:55.126634 27448 raft_consensus.cc:686] T 00000000000000000000000000000000 P ead6b34e9a5f408487edfe2e29e91eec [term 1 LEADER]: Becoming Leader. State: Replica: ead6b34e9a5f408487edfe2e29e91eec, State: Running, Role: LEADER
I20260620 19:13:55.126986 27448 consensus_queue.cc:237] T 00000000000000000000000000000000 P ead6b34e9a5f408487edfe2e29e91eec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ead6b34e9a5f408487edfe2e29e91eec" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 46025 } }
I20260620 19:13:55.127477 27443 sys_catalog.cc:564] T 00000000000000000000000000000000 P ead6b34e9a5f408487edfe2e29e91eec [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:13:55.128728 27450 sys_catalog.cc:455] T 00000000000000000000000000000000 P ead6b34e9a5f408487edfe2e29e91eec [sys.catalog]: SysCatalogTable state changed. Reason: New leader ead6b34e9a5f408487edfe2e29e91eec. Latest consensus state: current_term: 1 leader_uuid: "ead6b34e9a5f408487edfe2e29e91eec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ead6b34e9a5f408487edfe2e29e91eec" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 46025 } } }
I20260620 19:13:55.128729 27449 sys_catalog.cc:455] T 00000000000000000000000000000000 P ead6b34e9a5f408487edfe2e29e91eec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ead6b34e9a5f408487edfe2e29e91eec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ead6b34e9a5f408487edfe2e29e91eec" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 46025 } } }
I20260620 19:13:55.128877 27450 sys_catalog.cc:458] T 00000000000000000000000000000000 P ead6b34e9a5f408487edfe2e29e91eec [sys.catalog]: This master's current role is: LEADER
I20260620 19:13:55.128878 27449 sys_catalog.cc:458] T 00000000000000000000000000000000 P ead6b34e9a5f408487edfe2e29e91eec [sys.catalog]: This master's current role is: LEADER
I20260620 19:13:55.129307 27453 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:13:55.132699 27453 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:13:55.138526 27453 catalog_manager.cc:1347] Generated new cluster ID: 88b58378e13449acbd78fec541172573
I20260620 19:13:55.138603 27453 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:13:55.146641 27453 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:13:55.147614 27453 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:13:55.157770 27453 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ead6b34e9a5f408487edfe2e29e91eec: Generated new TSK 0
I20260620 19:13:55.158648 27453 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:13:55.166827 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982595231230-9837-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982595231230-9837-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982595231230-9837-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.65:0
--local_ip_for_outbound_sockets=127.9.155.65
--webserver_interface=127.9.155.65
--webserver_port=0
--tserver_master_addrs=127.9.155.126:46025
--builtin_ntp_servers=127.9.155.84:37773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 19:13:55.295401 27467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:13:55.295694 27467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:13:55.295733 27467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:13:55.299862 27467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:13:55.299937 27467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:13:55.299996 27467 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.65
I20260620 19:13:55.304935 27467 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:37773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982595231230-9837-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982595231230-9837-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.155.65
--webserver_port=0
--tserver_master_addrs=127.9.155.126:46025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27467
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.65
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982595231230-9837-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:13:55.306030 27467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:13:55.307444 27467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:13:55.317775 27473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:13:55.317873 27467 server_base.cc:1034] running on GCE node
W20260620 19:13:55.317797 27472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:13:55.317775 27475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:13:55.318598 27467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:13:55.319288 27467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:13:55.320447 27467 hybrid_clock.cc:648] HybridClock initialized: now 1781982835320444 us; error 38 us; skew 500 ppm
I20260620 19:13:55.322527 27467 webserver.cc:460] Webserver started at http://127.9.155.65:46375/ using document root <none> and password file <none>
I20260620 19:13:55.323169 27467 fs_manager.cc:359] Metadata directory not provided
I20260620 19:13:55.323251 27467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:13:55.323460 27467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:13:55.325645 27467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982595231230-9837-0/minicluster-data/ts-0/data/instance:
uuid: "17d28483166a415b8ffbce3bd354bf5f"
format_stamp: "Formatted at 2026-06-20 19:13:55 on dist-test-slave-n3fx"
server_key: "4f3a4d382aacc73436cfa91c0fdf0c24"
server_key_iv: "cc47bd948a58085a50f06a06f854c9ab"
server_key_version: "encryptionkey@0"
I20260620 19:13:55.326200 27467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982595231230-9837-0/minicluster-data/ts-0/wal/instance:
uuid: "17d28483166a415b8ffbce3bd354bf5f"
format_stamp: "Formatted at 2026-06-20 19:13:55 on dist-test-slave-n3fx"
server_key: "4f3a4d382aacc73436cfa91c0fdf0c24"
server_key_iv: "cc47bd948a58085a50f06a06f854c9ab"
server_key_version: "encryptionkey@0"
I20260620 19:13:55.330368 27467 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 19:13:55.333076 27481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:55.334429 27467 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:13:55.334558 27467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982595231230-9837-0/minicluster-data/ts-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982595231230-9837-0/minicluster-data/ts-0/wal
uuid: "17d28483166a415b8ffbce3bd354bf5f"
format_stamp: "Formatted at 2026-06-20 19:13:55 on dist-test-slave-n3fx"
server_key: "4f3a4d382aacc73436cfa91c0fdf0c24"
server_key_iv: "cc47bd948a58085a50f06a06f854c9ab"
server_key_version: "encryptionkey@0"
I20260620 19:13:55.334674 27467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982595231230-9837-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982595231230-9837-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982595231230-9837-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:13:55.366196 27467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:13:55.366931 27467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:13:55.367121 27467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:13:55.367702 27467 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:13:55.368664 27467 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:13:55.368710 27467 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:55.368762 27467 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:13:55.368790 27467 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:55.377645 27467 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.65:42193
I20260620 19:13:55.377703 27594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.65:42193 every 8 connection(s)
I20260620 19:13:55.378629 27467 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
I20260620 19:13:55.384471 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 27467
I20260620 19:13:55.384596 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982595231230-9837-0/minicluster-data/ts-0/wal/instance
I20260620 19:13:55.384881 9837 external_mini_cluster.cc:1428] Setting key 651067120086ed1e1ce5833625f5260e
I20260620 19:13:55.387077 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982595231230-9837-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982595231230-9837-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982595231230-9837-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.66:0
--local_ip_for_outbound_sockets=127.9.155.66
--webserver_interface=127.9.155.66
--webserver_port=0
--tserver_master_addrs=127.9.155.126:46025
--builtin_ntp_servers=127.9.155.84:37773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 19:13:55.389230 27446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:55.380563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.65:39023 (local address 127.9.155.126:46025)
0620 19:13:55.380688 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:55.380691 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:55.381868 (+ 1177us) server_negotiation.cc:407] Connection header received
0620 19:13:55.383337 (+ 1469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:55.383341 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:55.383412 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:55.383485 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:55.384598 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:55.385243 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:55.386709 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:55.387010 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:55.387109 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:55.388604 (+ 1495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:13:55.388609 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:13:55.388612 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:13:55.388629 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:13:55.388656 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:13:55.388666 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:13:55.388763 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:55.388985 (+ 222us) server_negotiation.cc:299] Negotiation successful
0620 19:13:55.389058 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 19:13:55.390192 27597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:55.380992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:46025 (local address 127.9.155.65:39023)
0620 19:13:55.381691 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:55.381735 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:55.383102 (+ 1367us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:55.383679 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:55.383687 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:55.383779 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:55.384408 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:55.384432 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:55.385398 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:55.385401 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:55.386555 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:55.386566 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:55.387210 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:55.387215 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:55.387473 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:55.388195 (+ 722us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:13:55.388197 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:55.388287 (+ 90us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:13:55.388403 (+ 116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:13:55.388786 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:13:55.388796 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:13:55.388801 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 19:13:55.388919 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 19:13:55.389194 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":112,"threads_started":1}
I20260620 19:13:55.391574 27595 heartbeater.cc:347] Connected to a master server at 127.9.155.126:46025
I20260620 19:13:55.391811 27595 heartbeater.cc:464] Registering TS with master...
I20260620 19:13:55.392390 27595 heartbeater.cc:511] Master 127.9.155.126:46025 requested a full tablet report, sending...
I20260620 19:13:55.394192 27407 ts_manager.cc:194] Registered new tserver with Master: 17d28483166a415b8ffbce3bd354bf5f (127.9.155.65:42193)
I20260620 19:13:55.395879 27407 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.155.65:39023
W20260620 19:13:55.500103 27598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:13:55.500473 27598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:13:55.500515 27598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:13:55.504700 27598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:13:55.504767 27598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:13:55.504825 27598 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.66
I20260620 19:13:55.509799 27598 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:37773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982595231230-9837-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982595231230-9837-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.155.66
--webserver_port=0
--tserver_master_addrs=127.9.155.126:46025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.66
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982595231230-9837-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:13:55.510978 27598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:13:55.512372 27598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:13:55.522318 27604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:13:55.522437 27606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:13:55.522318 27603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:13:55.522601 27598 server_base.cc:1034] running on GCE node
I20260620 19:13:55.523074 27598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:13:55.523792 27598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:13:55.524976 27598 hybrid_clock.cc:648] HybridClock initialized: now 1781982835524965 us; error 74 us; skew 500 ppm
I20260620 19:13:55.526942 27598 webserver.cc:460] Webserver started at http://127.9.155.66:45725/ using document root <none> and password file <none>
I20260620 19:13:55.527550 27598 fs_manager.cc:359] Metadata directory not provided
I20260620 19:13:55.527617 27598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:13:55.527799 27598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:13:55.529939 27598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982595231230-9837-0/minicluster-data/ts-1/data/instance:
uuid: "a74344cf828348b4ba7456e69baba3b9"
format_stamp: "Formatted at 2026-06-20 19:13:55 on dist-test-slave-n3fx"
server_key: "3df89312dc30ee69c7eb0634f006e568"
server_key_iv: "94e212e655c51272b90df141ff9cfce3"
server_key_version: "encryptionkey@0"
I20260620 19:13:55.530499 27598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982595231230-9837-0/minicluster-data/ts-1/wal/instance:
uuid: "a74344cf828348b4ba7456e69baba3b9"
format_stamp: "Formatted at 2026-06-20 19:13:55 on dist-test-slave-n3fx"
server_key: "3df89312dc30ee69c7eb0634f006e568"
server_key_iv: "94e212e655c51272b90df141ff9cfce3"
server_key_version: "encryptionkey@0"
I20260620 19:13:55.534433 27598 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260620 19:13:55.536944 27612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:55.538470 27598 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:13:55.538625 27598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982595231230-9837-0/minicluster-data/ts-1/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982595231230-9837-0/minicluster-data/ts-1/wal
uuid: "a74344cf828348b4ba7456e69baba3b9"
format_stamp: "Formatted at 2026-06-20 19:13:55 on dist-test-slave-n3fx"
server_key: "3df89312dc30ee69c7eb0634f006e568"
server_key_iv: "94e212e655c51272b90df141ff9cfce3"
server_key_version: "encryptionkey@0"
I20260620 19:13:55.538760 27598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982595231230-9837-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982595231230-9837-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982595231230-9837-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:13:55.571833 27598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:13:55.572606 27598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:13:55.572790 27598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:13:55.573374 27598 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:13:55.574347 27598 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:13:55.574404 27598 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:55.574458 27598 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:13:55.574487 27598 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:55.583307 27598 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.66:46307
I20260620 19:13:55.583338 27725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.66:46307 every 8 connection(s)
I20260620 19:13:55.584255 27598 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
I20260620 19:13:55.593569 27446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:55.586060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.66:32915 (local address 127.9.155.126:46025)
0620 19:13:55.586212 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:55.586216 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:55.587286 (+ 1070us) server_negotiation.cc:407] Connection header received
0620 19:13:55.588709 (+ 1423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:55.588712 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:55.588760 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:55.588832 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:55.589746 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:55.590290 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:55.591505 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:55.591730 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:55.591829 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:55.593114 (+ 1285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:13:55.593118 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:13:55.593120 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:13:55.593140 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:13:55.593162 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:13:55.593171 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:13:55.593233 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:55.593370 (+ 137us) server_negotiation.cc:299] Negotiation successful
0620 19:13:55.593428 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 19:13:55.594290 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 27598
I20260620 19:13:55.594381 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982595231230-9837-0/minicluster-data/ts-1/wal/instance
I20260620 19:13:55.594558 27728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:55.586399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:46025 (local address 127.9.155.66:32915)
0620 19:13:55.587118 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:55.587159 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:55.588517 (+ 1358us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:55.588976 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:55.588982 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:55.589062 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:55.589595 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:55.589609 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:55.590427 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:55.590430 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:55.591329 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:55.591339 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:55.591898 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:55.591902 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:55.592076 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:55.592787 (+ 711us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:13:55.592789 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:55.592832 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:13:55.592902 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:13:55.593252 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:13:55.593259 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:13:55.593264 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 19:13:55.593365 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 19:13:55.593631 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":118,"threads_started":1}
I20260620 19:13:55.594726 9837 external_mini_cluster.cc:1428] Setting key 17d2b938f61ac443edc12c1eda2ccf42
I20260620 19:13:55.595820 27726 heartbeater.cc:347] Connected to a master server at 127.9.155.126:46025
I20260620 19:13:55.596081 27726 heartbeater.cc:464] Registering TS with master...
I20260620 19:13:55.596738 27726 heartbeater.cc:511] Master 127.9.155.126:46025 requested a full tablet report, sending...
I20260620 19:13:55.597098 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982595231230-9837-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982595231230-9837-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982595231230-9837-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.67:0
--local_ip_for_outbound_sockets=127.9.155.67
--webserver_interface=127.9.155.67
--webserver_port=0
--tserver_master_addrs=127.9.155.126:46025
--builtin_ntp_servers=127.9.155.84:37773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 19:13:55.597836 27407 ts_manager.cc:194] Registered new tserver with Master: a74344cf828348b4ba7456e69baba3b9 (127.9.155.66:46307)
I20260620 19:13:55.598541 27407 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.155.66:32915
W20260620 19:13:55.707640 27729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:13:55.707926 27729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:13:55.707965 27729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:13:55.712102 27729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:13:55.712177 27729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:13:55.712273 27729 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.67
I20260620 19:13:55.717128 27729 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:37773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982595231230-9837-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982595231230-9837-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.155.67
--webserver_port=0
--tserver_master_addrs=127.9.155.126:46025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27729
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.67
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982595231230-9837-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:13:55.718276 27729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:13:55.719672 27729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:13:55.729857 27737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:13:55.729857 27734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:13:55.730088 27729 server_base.cc:1034] running on GCE node
W20260620 19:13:55.730159 27735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:13:55.730561 27729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:13:55.731261 27729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:13:55.732437 27729 hybrid_clock.cc:648] HybridClock initialized: now 1781982835732401 us; error 48 us; skew 500 ppm
I20260620 19:13:55.734373 27729 webserver.cc:460] Webserver started at http://127.9.155.67:45857/ using document root <none> and password file <none>
I20260620 19:13:55.734953 27729 fs_manager.cc:359] Metadata directory not provided
I20260620 19:13:55.735016 27729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:13:55.735209 27729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:13:55.737350 27729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982595231230-9837-0/minicluster-data/ts-2/data/instance:
uuid: "d56acf113bb9447c9d2f852e53167c88"
format_stamp: "Formatted at 2026-06-20 19:13:55 on dist-test-slave-n3fx"
server_key: "685ed667b2e7835ae9416ff0ec86ea40"
server_key_iv: "29f1b7e7cb51575b2034a374fdbcb9b1"
server_key_version: "encryptionkey@0"
I20260620 19:13:55.737977 27729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982595231230-9837-0/minicluster-data/ts-2/wal/instance:
uuid: "d56acf113bb9447c9d2f852e53167c88"
format_stamp: "Formatted at 2026-06-20 19:13:55 on dist-test-slave-n3fx"
server_key: "685ed667b2e7835ae9416ff0ec86ea40"
server_key_iv: "29f1b7e7cb51575b2034a374fdbcb9b1"
server_key_version: "encryptionkey@0"
I20260620 19:13:55.741969 27729 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 19:13:55.744545 27743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:55.745813 27729 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 19:13:55.745975 27729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982595231230-9837-0/minicluster-data/ts-2/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982595231230-9837-0/minicluster-data/ts-2/wal
uuid: "d56acf113bb9447c9d2f852e53167c88"
format_stamp: "Formatted at 2026-06-20 19:13:55 on dist-test-slave-n3fx"
server_key: "685ed667b2e7835ae9416ff0ec86ea40"
server_key_iv: "29f1b7e7cb51575b2034a374fdbcb9b1"
server_key_version: "encryptionkey@0"
I20260620 19:13:55.746089 27729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982595231230-9837-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982595231230-9837-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982595231230-9837-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:13:55.758183 27729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:13:55.758932 27729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:13:55.759114 27729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:13:55.759712 27729 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:13:55.760690 27729 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:13:55.760737 27729 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:55.760777 27729 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:13:55.760805 27729 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:55.769074 27729 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.67:36135
I20260620 19:13:55.769156 27856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.67:36135 every 8 connection(s)
I20260620 19:13:55.769979 27729 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
I20260620 19:13:55.773396 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 27729
I20260620 19:13:55.773500 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982595231230-9837-0/minicluster-data/ts-2/wal/instance
I20260620 19:13:55.773777 9837 external_mini_cluster.cc:1428] Setting key 4274fc4d98cda970c36b45dac6acc06a
I20260620 19:13:55.779446 27446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:55.771881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.67:60315 (local address 127.9.155.126:46025)
0620 19:13:55.772017 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:55.772020 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:55.773112 (+ 1092us) server_negotiation.cc:407] Connection header received
0620 19:13:55.774728 (+ 1616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:55.774731 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:55.774785 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:55.774859 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:55.775932 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:55.776488 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:55.777632 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:55.777853 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:55.777939 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:55.778998 (+ 1059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:13:55.779001 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:13:55.779003 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:13:55.779016 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:13:55.779034 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:13:55.779042 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:13:55.779100 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:55.779238 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 19:13:55.779302 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 19:13:55.780333 27859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:55.772295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:46025 (local address 127.9.155.67:60315)
0620 19:13:55.772936 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:55.772982 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:55.774518 (+ 1536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:55.775028 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:55.775035 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:55.775128 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:55.775802 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:55.775818 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:55.776614 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:55.776618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:55.777516 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:55.777526 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:55.778000 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:55.778004 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:55.778163 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:55.778738 (+ 575us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:13:55.778739 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:55.778780 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:13:55.778848 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:13:55.779110 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:13:55.779119 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:13:55.779125 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 19:13:55.779234 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 19:13:55.779526 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":113,"threads_started":1}
I20260620 19:13:55.781502 27857 heartbeater.cc:347] Connected to a master server at 127.9.155.126:46025
I20260620 19:13:55.781720 27857 heartbeater.cc:464] Registering TS with master...
I20260620 19:13:55.782207 27857 heartbeater.cc:511] Master 127.9.155.126:46025 requested a full tablet report, sending...
I20260620 19:13:55.783291 27407 ts_manager.cc:194] Registered new tserver with Master: d56acf113bb9447c9d2f852e53167c88 (127.9.155.67:36135)
I20260620 19:13:55.783867 27407 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.155.67:60315
I20260620 19:13:55.787664 9837 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:13:55.792968 27446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:55.789561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49978 (local address 127.9.155.126:46025)
0620 19:13:55.789690 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:55.789694 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:55.789808 (+ 114us) server_negotiation.cc:407] Connection header received
0620 19:13:55.789946 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:55.789948 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:55.789985 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:55.790053 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:55.790279 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:55.790784 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:55.791601 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:55.791808 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:55.791931 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:55.792602 (+ 671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:13:55.792605 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:13:55.792608 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:13:55.792618 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:13:55.792636 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:13:55.792644 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:13:55.792698 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:55.792767 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 19:13:55.792814 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 19:13:55.795583 27407 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49978:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:13:55.798520 27407 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:13:55.811694 27868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:55.806251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.67:36135 (local address 127.0.0.1:52302)
0620 19:13:55.807005 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:55.807058 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:55.807183 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:55.807705 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:55.807711 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:55.807734 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:13:55.807911 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:55.807918 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:55.809209 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:55.809212 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:55.810354 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:55.810363 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:55.811113 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:55.811116 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:55.811275 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:55.811317 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 19:13:55.811422 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 19:13:55.811539 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"mutex_wait_us":24,"spinlock_wait_cycles":5376,"thread_start_us":69,"threads_started":1}
I20260620 19:13:55.811697 27870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:55.806379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52302 (local address 127.9.155.67:36135)
0620 19:13:55.807212 (+ 833us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:55.807224 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:55.807250 (+ 26us) server_negotiation.cc:407] Connection header received
0620 19:13:55.807323 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:55.807328 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:55.807530 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:55.808052 (+ 522us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:13:55.808183 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:55.809103 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:55.810461 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:55.810997 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:55.811092 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:55.811287 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:55.811460 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 19:13:55.811555 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":729,"thread_start_us":146,"threads_started":1}
I20260620 19:13:55.812606 27872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:55.807129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.65:42193 (local address 127.0.0.1:45618)
0620 19:13:55.807689 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:55.807702 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:55.807789 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:55.808289 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:55.808292 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:55.808308 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:13:55.808418 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:55.808424 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:55.809831 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:55.809834 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:55.811190 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:55.811202 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:55.812236 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:55.812241 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:55.812328 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:55.812358 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 19:13:55.812420 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 19:13:55.812481 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"mutex_wait_us":104,"thread_start_us":82,"threads_started":1}
I20260620 19:13:55.812822 27873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:55.807520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45618 (local address 127.9.155.65:42193)
0620 19:13:55.807932 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:55.807941 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:55.807957 (+ 16us) server_negotiation.cc:407] Connection header received
0620 19:13:55.808015 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:55.808019 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:55.808149 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:55.808303 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:13:55.808522 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:55.809695 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:55.811328 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:55.812106 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:55.812209 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:55.812406 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:55.812453 (+ 47us) server_negotiation.cc:299] Negotiation successful
0620 19:13:55.812542 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":106,"threads_started":1}
I20260620 19:13:55.813030 27869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:55.806353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.66:46307 (local address 127.0.0.1:45264)
0620 19:13:55.807629 (+ 1276us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:55.807641 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:55.807771 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:55.808414 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:55.808417 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:55.808434 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:13:55.808515 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:55.808521 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:55.809846 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:55.809849 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:55.811226 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:55.811232 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:55.812672 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:55.812675 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:55.812759 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:55.812780 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 19:13:55.812842 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 19:13:55.812905 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1215,"mutex_wait_us":121,"spinlock_wait_cycles":384,"thread_start_us":95,"threads_started":1}
I20260620 19:13:55.813476 27871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:55.806559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45264 (local address 127.9.155.66:46307)
0620 19:13:55.807480 (+ 921us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:55.807490 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:55.807710 (+ 220us) server_negotiation.cc:407] Connection header received
0620 19:13:55.807904 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:55.807910 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:55.808093 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:55.808203 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:13:55.808627 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:55.809744 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:55.811342 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:55.812349 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:55.812456 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:55.812610 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:55.813222 (+ 612us) server_negotiation.cc:299] Negotiation successful
0620 19:13:55.813313 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":821,"thread_start_us":93,"threads_started":1}
I20260620 19:13:55.814779 27791 tablet_service.cc:1467] Processing CreateTablet for tablet 355dedc333f54e82afc3d913fb024086 (DEFAULT_TABLE table=test-table [id=32d06d95fc1c4939a5042ea9c33f8ee1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:13:55.815726 27791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 355dedc333f54e82afc3d913fb024086. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:13:55.815555 27529 tablet_service.cc:1467] Processing CreateTablet for tablet 355dedc333f54e82afc3d913fb024086 (DEFAULT_TABLE table=test-table [id=32d06d95fc1c4939a5042ea9c33f8ee1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:13:55.815943 27660 tablet_service.cc:1467] Processing CreateTablet for tablet 355dedc333f54e82afc3d913fb024086 (DEFAULT_TABLE table=test-table [id=32d06d95fc1c4939a5042ea9c33f8ee1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:13:55.816586 27529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 355dedc333f54e82afc3d913fb024086. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:13:55.816900 27660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 355dedc333f54e82afc3d913fb024086. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:13:55.822934 27874 tablet_bootstrap.cc:492] T 355dedc333f54e82afc3d913fb024086 P d56acf113bb9447c9d2f852e53167c88: Bootstrap starting.
I20260620 19:13:55.824252 27875 tablet_bootstrap.cc:492] T 355dedc333f54e82afc3d913fb024086 P a74344cf828348b4ba7456e69baba3b9: Bootstrap starting.
I20260620 19:13:55.824739 27876 tablet_bootstrap.cc:492] T 355dedc333f54e82afc3d913fb024086 P 17d28483166a415b8ffbce3bd354bf5f: Bootstrap starting.
I20260620 19:13:55.825187 27874 tablet_bootstrap.cc:654] T 355dedc333f54e82afc3d913fb024086 P d56acf113bb9447c9d2f852e53167c88: Neither blocks nor log segments found. Creating new log.
I20260620 19:13:55.825902 27874 log.cc:824] T 355dedc333f54e82afc3d913fb024086 P d56acf113bb9447c9d2f852e53167c88: Log is configured to *not* fsync() on all Append() calls
I20260620 19:13:55.826476 27875 tablet_bootstrap.cc:654] T 355dedc333f54e82afc3d913fb024086 P a74344cf828348b4ba7456e69baba3b9: Neither blocks nor log segments found. Creating new log.
I20260620 19:13:55.826779 27876 tablet_bootstrap.cc:654] T 355dedc333f54e82afc3d913fb024086 P 17d28483166a415b8ffbce3bd354bf5f: Neither blocks nor log segments found. Creating new log.
I20260620 19:13:55.827344 27875 log.cc:824] T 355dedc333f54e82afc3d913fb024086 P a74344cf828348b4ba7456e69baba3b9: Log is configured to *not* fsync() on all Append() calls
I20260620 19:13:55.827539 27876 log.cc:824] T 355dedc333f54e82afc3d913fb024086 P 17d28483166a415b8ffbce3bd354bf5f: Log is configured to *not* fsync() on all Append() calls
I20260620 19:13:55.827759 27874 tablet_bootstrap.cc:492] T 355dedc333f54e82afc3d913fb024086 P d56acf113bb9447c9d2f852e53167c88: No bootstrap required, opened a new log
I20260620 19:13:55.827950 27874 ts_tablet_manager.cc:1397] T 355dedc333f54e82afc3d913fb024086 P d56acf113bb9447c9d2f852e53167c88: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 19:13:55.829706 27876 tablet_bootstrap.cc:492] T 355dedc333f54e82afc3d913fb024086 P 17d28483166a415b8ffbce3bd354bf5f: No bootstrap required, opened a new log
I20260620 19:13:55.829893 27876 ts_tablet_manager.cc:1397] T 355dedc333f54e82afc3d913fb024086 P 17d28483166a415b8ffbce3bd354bf5f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 19:13:55.829969 27875 tablet_bootstrap.cc:492] T 355dedc333f54e82afc3d913fb024086 P a74344cf828348b4ba7456e69baba3b9: No bootstrap required, opened a new log
I20260620 19:13:55.830156 27875 ts_tablet_manager.cc:1397] T 355dedc333f54e82afc3d913fb024086 P a74344cf828348b4ba7456e69baba3b9: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260620 19:13:55.832693 27874 raft_consensus.cc:348] T 355dedc333f54e82afc3d913fb024086 P d56acf113bb9447c9d2f852e53167c88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d56acf113bb9447c9d2f852e53167c88" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 36135 } } peers { permanent_uuid: "a74344cf828348b4ba7456e69baba3b9" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 46307 } } peers { permanent_uuid: "17d28483166a415b8ffbce3bd354bf5f" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 42193 } }
I20260620 19:13:55.832990 27874 raft_consensus.cc:374] T 355dedc333f54e82afc3d913fb024086 P d56acf113bb9447c9d2f852e53167c88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:13:55.833050 27874 raft_consensus.cc:729] T 355dedc333f54e82afc3d913fb024086 P d56acf113bb9447c9d2f852e53167c88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d56acf113bb9447c9d2f852e53167c88, State: Initialized, Role: FOLLOWER
I20260620 19:13:55.833110 27876 raft_consensus.cc:348] T 355dedc333f54e82afc3d913fb024086 P 17d28483166a415b8ffbce3bd354bf5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d56acf113bb9447c9d2f852e53167c88" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 36135 } } peers { permanent_uuid: "a74344cf828348b4ba7456e69baba3b9" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 46307 } } peers { permanent_uuid: "17d28483166a415b8ffbce3bd354bf5f" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 42193 } }
I20260620 19:13:55.833329 27876 raft_consensus.cc:374] T 355dedc333f54e82afc3d913fb024086 P 17d28483166a415b8ffbce3bd354bf5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:13:55.833391 27876 raft_consensus.cc:729] T 355dedc333f54e82afc3d913fb024086 P 17d28483166a415b8ffbce3bd354bf5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17d28483166a415b8ffbce3bd354bf5f, State: Initialized, Role: FOLLOWER
I20260620 19:13:55.833576 27874 consensus_queue.cc:260] T 355dedc333f54e82afc3d913fb024086 P d56acf113bb9447c9d2f852e53167c88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d56acf113bb9447c9d2f852e53167c88" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 36135 } } peers { permanent_uuid: "a74344cf828348b4ba7456e69baba3b9" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 46307 } } peers { permanent_uuid: "17d28483166a415b8ffbce3bd354bf5f" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 42193 } }
I20260620 19:13:55.833827 27876 consensus_queue.cc:260] T 355dedc333f54e82afc3d913fb024086 P 17d28483166a415b8ffbce3bd354bf5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d56acf113bb9447c9d2f852e53167c88" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 36135 } } peers { permanent_uuid: "a74344cf828348b4ba7456e69baba3b9" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 46307 } } peers { permanent_uuid: "17d28483166a415b8ffbce3bd354bf5f" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 42193 } }
I20260620 19:13:55.833906 27875 raft_consensus.cc:348] T 355dedc333f54e82afc3d913fb024086 P a74344cf828348b4ba7456e69baba3b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d56acf113bb9447c9d2f852e53167c88" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 36135 } } peers { permanent_uuid: "a74344cf828348b4ba7456e69baba3b9" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 46307 } } peers { permanent_uuid: "17d28483166a415b8ffbce3bd354bf5f" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 42193 } }
I20260620 19:13:55.834148 27875 raft_consensus.cc:374] T 355dedc333f54e82afc3d913fb024086 P a74344cf828348b4ba7456e69baba3b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:13:55.834198 27875 raft_consensus.cc:729] T 355dedc333f54e82afc3d913fb024086 P a74344cf828348b4ba7456e69baba3b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a74344cf828348b4ba7456e69baba3b9, State: Initialized, Role: FOLLOWER
I20260620 19:13:55.834482 27857 heartbeater.cc:503] Master 127.9.155.126:46025 was elected leader, sending a full tablet report...
I20260620 19:13:55.834671 27874 ts_tablet_manager.cc:1428] T 355dedc333f54e82afc3d913fb024086 P d56acf113bb9447c9d2f852e53167c88: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260620 19:13:55.834643 27875 consensus_queue.cc:260] T 355dedc333f54e82afc3d913fb024086 P a74344cf828348b4ba7456e69baba3b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d56acf113bb9447c9d2f852e53167c88" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 36135 } } peers { permanent_uuid: "a74344cf828348b4ba7456e69baba3b9" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 46307 } } peers { permanent_uuid: "17d28483166a415b8ffbce3bd354bf5f" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 42193 } }
I20260620 19:13:55.834730 27876 ts_tablet_manager.cc:1428] T 355dedc333f54e82afc3d913fb024086 P 17d28483166a415b8ffbce3bd354bf5f: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 19:13:55.835047 27595 heartbeater.cc:503] Master 127.9.155.126:46025 was elected leader, sending a full tablet report...
I20260620 19:13:55.835291 27726 heartbeater.cc:503] Master 127.9.155.126:46025 was elected leader, sending a full tablet report...
I20260620 19:13:55.836257 27875 ts_tablet_manager.cc:1428] T 355dedc333f54e82afc3d913fb024086 P a74344cf828348b4ba7456e69baba3b9: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260620 19:13:55.879827 27596 tablet.cc:2369] T 355dedc333f54e82afc3d913fb024086 P 17d28483166a415b8ffbce3bd354bf5f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:13:55.968747 27881 raft_consensus.cc:482] T 355dedc333f54e82afc3d913fb024086 P 17d28483166a415b8ffbce3bd354bf5f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:13:55.969055 27881 raft_consensus.cc:504] T 355dedc333f54e82afc3d913fb024086 P 17d28483166a415b8ffbce3bd354bf5f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d56acf113bb9447c9d2f852e53167c88" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 36135 } } peers { permanent_uuid: "a74344cf828348b4ba7456e69baba3b9" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 46307 } } peers { permanent_uuid: "17d28483166a415b8ffbce3bd354bf5f" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 42193 } }
I20260620 19:13:55.970288 27881 leader_election.cc:288] T 355dedc333f54e82afc3d913fb024086 P 17d28483166a415b8ffbce3bd354bf5f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d56acf113bb9447c9d2f852e53167c88 (127.9.155.67:36135), a74344cf828348b4ba7456e69baba3b9 (127.9.155.66:46307)
I20260620 19:13:55.974238 27871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:55.970731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.65:52633 (local address 127.9.155.66:46307)
0620 19:13:55.970872 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:55.970877 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:55.971176 (+ 299us) server_negotiation.cc:407] Connection header received
0620 19:13:55.971292 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:55.971295 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:55.971341 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:55.971406 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:13:55.971675 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:55.972181 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:55.973331 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:55.973715 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:55.973826 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:55.973879 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:55.974041 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 19:13:55.974115 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 19:13:55.974282 27884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:55.970608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.66:46307 (local address 127.9.155.65:52633)
0620 19:13:55.971078 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:55.971090 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:55.971193 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:55.971435 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:55.971438 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:55.971454 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:13:55.971569 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:55.971575 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:55.972321 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:55.972325 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:55.973212 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:55.973221 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:55.973826 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:55.973830 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:55.973931 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:55.973951 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 19:13:55.974013 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 19:13:55.974088 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"mutex_wait_us":79,"thread_start_us":129,"threads_started":1}
I20260620 19:13:55.974279 27870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:55.970759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.65:40277 (local address 127.9.155.67:36135)
0620 19:13:55.970984 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:55.970988 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:55.971066 (+ 78us) server_negotiation.cc:407] Connection header received
0620 19:13:55.971219 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:55.971221 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:55.971264 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:55.971327 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:13:55.971611 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:55.972132 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:55.973323 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:55.973741 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:55.973851 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:55.973902 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:55.974080 (+ 178us) server_negotiation.cc:299] Negotiation successful
0620 19:13:55.974151 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
I20260620 19:13:55.974632 27883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:55.970600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.67:36135 (local address 127.9.155.65:40277)
0620 19:13:55.970961 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:55.970977 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:55.971109 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:55.971355 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:55.971357 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:55.971373 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:13:55.971503 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:55.971509 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:55.972272 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:55.972275 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:55.973199 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:55.973208 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:55.973862 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:55.973865 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:55.973969 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:55.973987 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 19:13:55.974050 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 19:13:55.974133 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"mutex_wait_us":51,"thread_start_us":87,"threads_started":1}
I20260620 19:13:55.974928 27680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "355dedc333f54e82afc3d913fb024086" candidate_uuid: "17d28483166a415b8ffbce3bd354bf5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a74344cf828348b4ba7456e69baba3b9" is_pre_election: true
I20260620 19:13:55.975286 27680 raft_consensus.cc:2454] T 355dedc333f54e82afc3d913fb024086 P a74344cf828348b4ba7456e69baba3b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17d28483166a415b8ffbce3bd354bf5f in term 0.
I20260620 19:13:55.975261 27811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "355dedc333f54e82afc3d913fb024086" candidate_uuid: "17d28483166a415b8ffbce3bd354bf5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d56acf113bb9447c9d2f852e53167c88" is_pre_election: true
I20260620 19:13:55.975550 27811 raft_consensus.cc:2454] T 355dedc333f54e82afc3d913fb024086 P d56acf113bb9447c9d2f852e53167c88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17d28483166a415b8ffbce3bd354bf5f in term 0.
I20260620 19:13:55.975777 27483 leader_election.cc:302] T 355dedc333f54e82afc3d913fb024086 P 17d28483166a415b8ffbce3bd354bf5f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17d28483166a415b8ffbce3bd354bf5f, a74344cf828348b4ba7456e69baba3b9; no voters:
I20260620 19:13:55.976094 27881 raft_consensus.cc:2781] T 355dedc333f54e82afc3d913fb024086 P 17d28483166a415b8ffbce3bd354bf5f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:13:55.976194 27881 raft_consensus.cc:482] T 355dedc333f54e82afc3d913fb024086 P 17d28483166a415b8ffbce3bd354bf5f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:13:55.976265 27881 raft_consensus.cc:3037] T 355dedc333f54e82afc3d913fb024086 P 17d28483166a415b8ffbce3bd354bf5f [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:13:55.977311 27881 raft_consensus.cc:504] T 355dedc333f54e82afc3d913fb024086 P 17d28483166a415b8ffbce3bd354bf5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d56acf113bb9447c9d2f852e53167c88" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 36135 } } peers { permanent_uuid: "a74344cf828348b4ba7456e69baba3b9" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 46307 } } peers { permanent_uuid: "17d28483166a415b8ffbce3bd354bf5f" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 42193 } }
I20260620 19:13:55.977792 27881 leader_election.cc:288] T 355dedc333f54e82afc3d913fb024086 P 17d28483166a415b8ffbce3bd354bf5f [CANDIDATE]: Term 1 election: Requested vote from peers d56acf113bb9447c9d2f852e53167c88 (127.9.155.67:36135), a74344cf828348b4ba7456e69baba3b9 (127.9.155.66:46307)
I20260620 19:13:55.978193 27811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "355dedc333f54e82afc3d913fb024086" candidate_uuid: "17d28483166a415b8ffbce3bd354bf5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d56acf113bb9447c9d2f852e53167c88"
I20260620 19:13:55.978188 27680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "355dedc333f54e82afc3d913fb024086" candidate_uuid: "17d28483166a415b8ffbce3bd354bf5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a74344cf828348b4ba7456e69baba3b9"
I20260620 19:13:55.978324 27811 raft_consensus.cc:3037] T 355dedc333f54e82afc3d913fb024086 P d56acf113bb9447c9d2f852e53167c88 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:13:55.978320 27680 raft_consensus.cc:3037] T 355dedc333f54e82afc3d913fb024086 P a74344cf828348b4ba7456e69baba3b9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:13:55.979391 27680 raft_consensus.cc:2454] T 355dedc333f54e82afc3d913fb024086 P a74344cf828348b4ba7456e69baba3b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17d28483166a415b8ffbce3bd354bf5f in term 1.
I20260620 19:13:55.979444 27811 raft_consensus.cc:2454] T 355dedc333f54e82afc3d913fb024086 P d56acf113bb9447c9d2f852e53167c88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17d28483166a415b8ffbce3bd354bf5f in term 1.
I20260620 19:13:55.979779 27484 leader_election.cc:302] T 355dedc333f54e82afc3d913fb024086 P 17d28483166a415b8ffbce3bd354bf5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17d28483166a415b8ffbce3bd354bf5f, d56acf113bb9447c9d2f852e53167c88; no voters:
I20260620 19:13:55.979955 27881 raft_consensus.cc:2781] T 355dedc333f54e82afc3d913fb024086 P 17d28483166a415b8ffbce3bd354bf5f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:13:55.980183 27881 raft_consensus.cc:686] T 355dedc333f54e82afc3d913fb024086 P 17d28483166a415b8ffbce3bd354bf5f [term 1 LEADER]: Becoming Leader. State: Replica: 17d28483166a415b8ffbce3bd354bf5f, State: Running, Role: LEADER
I20260620 19:13:55.980612 27881 consensus_queue.cc:237] T 355dedc333f54e82afc3d913fb024086 P 17d28483166a415b8ffbce3bd354bf5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d56acf113bb9447c9d2f852e53167c88" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 36135 } } peers { permanent_uuid: "a74344cf828348b4ba7456e69baba3b9" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 46307 } } peers { permanent_uuid: "17d28483166a415b8ffbce3bd354bf5f" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 42193 } }
I20260620 19:13:55.983873 27406 catalog_manager.cc:5495] T 355dedc333f54e82afc3d913fb024086 P 17d28483166a415b8ffbce3bd354bf5f reported cstate change: term changed from 0 to 1, leader changed from <none> to 17d28483166a415b8ffbce3bd354bf5f (127.9.155.65). New cstate: current_term: 1 leader_uuid: "17d28483166a415b8ffbce3bd354bf5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d56acf113bb9447c9d2f852e53167c88" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 36135 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a74344cf828348b4ba7456e69baba3b9" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 46307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17d28483166a415b8ffbce3bd354bf5f" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 42193 } health_report { overall_health: HEALTHY } } }
I20260620 19:13:56.006139 27873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:56.002427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45626 (local address 127.9.155.65:42193)
0620 19:13:56.002544 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:56.002547 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:56.002559 (+ 12us) server_negotiation.cc:407] Connection header received
0620 19:13:56.002635 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:56.002638 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:56.002698 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:56.002791 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:13:56.003130 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:56.003660 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:56.004632 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:56.004853 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:56.004937 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:56.005290 (+ 353us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:13:56.005753 (+ 463us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:13:56.005845 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:56.005955 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 19:13:56.006012 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 19:13:56.012092 27680 raft_consensus.cc:1261] T 355dedc333f54e82afc3d913fb024086 P a74344cf828348b4ba7456e69baba3b9 [term 1 FOLLOWER]: Refusing update from remote peer 17d28483166a415b8ffbce3bd354bf5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:13:56.012092 27811 raft_consensus.cc:1261] T 355dedc333f54e82afc3d913fb024086 P d56acf113bb9447c9d2f852e53167c88 [term 1 FOLLOWER]: Refusing update from remote peer 17d28483166a415b8ffbce3bd354bf5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:13:56.012785 27881 consensus_queue.cc:1035] T 355dedc333f54e82afc3d913fb024086 P 17d28483166a415b8ffbce3bd354bf5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d56acf113bb9447c9d2f852e53167c88" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 36135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:13:56.012984 27885 consensus_queue.cc:1035] T 355dedc333f54e82afc3d913fb024086 P 17d28483166a415b8ffbce3bd354bf5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74344cf828348b4ba7456e69baba3b9" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 46307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:13:56.021131 27889 mvcc.cc:205] Tried to move back new op lower bound from 7299001696298299392 to 7299001696179412992. Current Snapshot: MvccSnapshot[applied={T|T < 7299001696298299392}]
I20260620 19:13:56.021277 27890 mvcc.cc:205] Tried to move back new op lower bound from 7299001696298299392 to 7299001696179412992. Current Snapshot: MvccSnapshot[applied={T|T < 7299001696298299392}]
I20260620 19:13:56.029775 27406 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49978:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:13:56.029948 27406 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49978:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:13:56.032757 27406 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ead6b34e9a5f408487edfe2e29e91eec: Sending DeleteTablet for 3 replicas of tablet 355dedc333f54e82afc3d913fb024086
I20260620 19:13:56.033707 27529 tablet_service.cc:1514] Processing DeleteTablet for tablet 355dedc333f54e82afc3d913fb024086 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:13:56 UTC) from {username='slave'} at 127.0.0.1:45618
I20260620 19:13:56.033687 27791 tablet_service.cc:1514] Processing DeleteTablet for tablet 355dedc333f54e82afc3d913fb024086 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:13:56 UTC) from {username='slave'} at 127.0.0.1:52302
I20260620 19:13:56.034202 27898 tablet_replica.cc:330] T 355dedc333f54e82afc3d913fb024086 P 17d28483166a415b8ffbce3bd354bf5f: stopping tablet replica
I20260620 19:13:56.034211 27660 tablet_service.cc:1514] Processing DeleteTablet for tablet 355dedc333f54e82afc3d913fb024086 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:13:56 UTC) from {username='slave'} at 127.0.0.1:45264
I20260620 19:13:56.034598 27898 raft_consensus.cc:2229] T 355dedc333f54e82afc3d913fb024086 P 17d28483166a415b8ffbce3bd354bf5f [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:13:56.034686 27899 tablet_replica.cc:330] T 355dedc333f54e82afc3d913fb024086 P d56acf113bb9447c9d2f852e53167c88: stopping tablet replica
I20260620 19:13:56.034950 27899 raft_consensus.cc:2229] T 355dedc333f54e82afc3d913fb024086 P d56acf113bb9447c9d2f852e53167c88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:13:56.034991 27898 raft_consensus.cc:2258] T 355dedc333f54e82afc3d913fb024086 P 17d28483166a415b8ffbce3bd354bf5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:13:56.035337 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 27467
I20260620 19:13:56.035378 27899 raft_consensus.cc:2258] T 355dedc333f54e82afc3d913fb024086 P d56acf113bb9447c9d2f852e53167c88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:13:56.037552 27899 ts_tablet_manager.cc:1905] T 355dedc333f54e82afc3d913fb024086 P d56acf113bb9447c9d2f852e53167c88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:13:56.037618 27900 tablet_replica.cc:330] T 355dedc333f54e82afc3d913fb024086 P a74344cf828348b4ba7456e69baba3b9: stopping tablet replica
I20260620 19:13:56.037868 27900 raft_consensus.cc:2229] T 355dedc333f54e82afc3d913fb024086 P a74344cf828348b4ba7456e69baba3b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:13:56.038147 27900 raft_consensus.cc:2258] T 355dedc333f54e82afc3d913fb024086 P a74344cf828348b4ba7456e69baba3b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:13:56.039319 27900 ts_tablet_manager.cc:1905] T 355dedc333f54e82afc3d913fb024086 P a74344cf828348b4ba7456e69baba3b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:13:56.040238 27899 ts_tablet_manager.cc:1918] T 355dedc333f54e82afc3d913fb024086 P d56acf113bb9447c9d2f852e53167c88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:13:56.040331 27899 log.cc:1196] T 355dedc333f54e82afc3d913fb024086 P d56acf113bb9447c9d2f852e53167c88: Deleting WAL directory at /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982595231230-9837-0/minicluster-data/ts-2/wal/wals/355dedc333f54e82afc3d913fb024086
I20260620 19:13:56.040608 27899 ts_tablet_manager.cc:1939] T 355dedc333f54e82afc3d913fb024086 P d56acf113bb9447c9d2f852e53167c88: Deleting consensus metadata
I20260620 19:13:56.041497 27900 ts_tablet_manager.cc:1918] T 355dedc333f54e82afc3d913fb024086 P a74344cf828348b4ba7456e69baba3b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:13:56.041577 27900 log.cc:1196] T 355dedc333f54e82afc3d913fb024086 P a74344cf828348b4ba7456e69baba3b9: Deleting WAL directory at /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982595231230-9837-0/minicluster-data/ts-1/wal/wals/355dedc333f54e82afc3d913fb024086
I20260620 19:13:56.041869 27900 ts_tablet_manager.cc:1939] T 355dedc333f54e82afc3d913fb024086 P a74344cf828348b4ba7456e69baba3b9: Deleting consensus metadata
I20260620 19:13:56.041874 27393 catalog_manager.cc:4841] TS d56acf113bb9447c9d2f852e53167c88 (127.9.155.67:36135): tablet 355dedc333f54e82afc3d913fb024086 (table test-table [id=32d06d95fc1c4939a5042ea9c33f8ee1]) successfully deleted
W20260620 19:13:56.042346 27392 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.155.65:42193 (error 108)
I20260620 19:13:56.042569 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 27598
I20260620 19:13:56.043673 27869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:56.043406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.65:42193 (local address 127.0.0.1:45640)
0620 19:13:56.043537 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:56.043601 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.65:42193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 19:13:56.043807 27392 catalog_manager.cc:4584] TS 17d28483166a415b8ffbce3bd354bf5f (127.9.155.65:42193): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 355dedc333f54e82afc3d913fb024086: Network error: Client connection negotiation failed: client connection to 127.9.155.65:42193: connect: Connection refused (error 111)
I20260620 19:13:56.047544 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 27729
I20260620 19:13:56.047986 27869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:56.047700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.66:46307 (local address 127.0.0.1:45266)
0620 19:13:56.047853 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:56.047896 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.66:46307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260620 19:13:56.053094 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 27376
2026-06-20T19:13:56Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1144 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:13:56 dist-test-slave-n3fx krb5kdc[27904](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:13:56 dist-test-slave-n3fx krb5kdc[27904](info): set up 2 sockets
Jun 20 19:13:56 dist-test-slave-n3fx krb5kdc[27904](info): commencing operation
krb5kdc: starting...
W20260620 19:13:57.012166 9837 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/krb5kdc/test-user.keytab.
Jun 20 19:13:57 dist-test-slave-n3fx krb5kdc[27904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:13:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:13:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.155.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.155.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.155.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab.
Entry for principal kudu/127.9.155.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab.
I20260620 19:13:57.131461 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:37887
--webserver_interface=127.9.155.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab
--principal=kudu/127.9.155.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:37861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.155.126:37887
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:13:57.239928 27917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:13:57.240200 27917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:13:57.240259 27917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:13:57.244249 27917 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:13:57.244320 27917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:13:57.244339 27917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:13:57.244354 27917 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:13:57.244369 27917 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:13:57.249763 27917 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:37861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.155.126:37887
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab
--principal=kudu/127.9.155.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:37887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.155.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27917
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:13:57.250808 27917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:13:57.251803 27917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:13:57.260123 27923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:13:57.260159 27922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:13:57.260164 27925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:13:57.260247 27917 server_base.cc:1034] running on GCE node
I20260620 19:13:57.260986 27917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:13:57.262045 27917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:13:57.263206 27917 hybrid_clock.cc:648] HybridClock initialized: now 1781982837263191 us; error 30 us; skew 500 ppm
Jun 20 19:13:57 dist-test-slave-n3fx krb5kdc[27904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982837, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:13:57.266039 27917 init.cc:377] Logged in from keytab as kudu/127.9.155.126@KRBTEST.COM (short username kudu)
I20260620 19:13:57.266923 27917 webserver.cc:460] Webserver started at http://127.9.155.126:33979/ using document root <none> and password file <none>
I20260620 19:13:57.267462 27917 fs_manager.cc:359] Metadata directory not provided
I20260620 19:13:57.267527 27917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:13:57.267719 27917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:13:57.269713 27917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/minicluster-data/master-0/data/instance:
uuid: "8f71d2d3ddc34905bf4ca9cc877d4e86"
format_stamp: "Formatted at 2026-06-20 19:13:57 on dist-test-slave-n3fx"
server_key: "3ad671b0a6b23b2d02c5529a5dd1b0fa"
server_key_iv: "ae19597efec562b7c1c57a90c82f6658"
server_key_version: "encryptionkey@0"
I20260620 19:13:57.270238 27917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/minicluster-data/master-0/wal/instance:
uuid: "8f71d2d3ddc34905bf4ca9cc877d4e86"
format_stamp: "Formatted at 2026-06-20 19:13:57 on dist-test-slave-n3fx"
server_key: "3ad671b0a6b23b2d02c5529a5dd1b0fa"
server_key_iv: "ae19597efec562b7c1c57a90c82f6658"
server_key_version: "encryptionkey@0"
I20260620 19:13:57.274003 27917 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 19:13:57.276372 27932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:57.277707 27917 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 19:13:57.277833 27917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/minicluster-data/master-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/minicluster-data/master-0/wal
uuid: "8f71d2d3ddc34905bf4ca9cc877d4e86"
format_stamp: "Formatted at 2026-06-20 19:13:57 on dist-test-slave-n3fx"
server_key: "3ad671b0a6b23b2d02c5529a5dd1b0fa"
server_key_iv: "ae19597efec562b7c1c57a90c82f6658"
server_key_version: "encryptionkey@0"
I20260620 19:13:57.277937 27917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:13:57.287878 27917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:13:57.290725 27917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:13:57.290906 27917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:13:57.298434 27917 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.126:37887
I20260620 19:13:57.298473 27984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.126:37887 every 8 connection(s)
I20260620 19:13:57.299381 27917 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
I20260620 19:13:57.302191 27985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:13:57.307569 27985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f71d2d3ddc34905bf4ca9cc877d4e86: Bootstrap starting.
I20260620 19:13:57.307742 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 27917
I20260620 19:13:57.307870 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/minicluster-data/master-0/wal/instance
I20260620 19:13:57.308115 9837 external_mini_cluster.cc:1428] Setting key 10fc5b9a8c98110728ef78b077fb9ad0
I20260620 19:13:57.310007 27985 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f71d2d3ddc34905bf4ca9cc877d4e86: Neither blocks nor log segments found. Creating new log.
I20260620 19:13:57.310730 27985 log.cc:824] T 00000000000000000000000000000000 P 8f71d2d3ddc34905bf4ca9cc877d4e86: Log is configured to *not* fsync() on all Append() calls
I20260620 19:13:57.312937 27985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f71d2d3ddc34905bf4ca9cc877d4e86: No bootstrap required, opened a new log
Jun 20 19:13:57 dist-test-slave-n3fx krb5kdc[27904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:13:57.317317 27985 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f71d2d3ddc34905bf4ca9cc877d4e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f71d2d3ddc34905bf4ca9cc877d4e86" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 37887 } }
I20260620 19:13:57.317571 27985 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f71d2d3ddc34905bf4ca9cc877d4e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:13:57.317652 27985 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f71d2d3ddc34905bf4ca9cc877d4e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f71d2d3ddc34905bf4ca9cc877d4e86, State: Initialized, Role: FOLLOWER
I20260620 19:13:57.318142 27985 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f71d2d3ddc34905bf4ca9cc877d4e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f71d2d3ddc34905bf4ca9cc877d4e86" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 37887 } }
I20260620 19:13:57.318275 27985 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f71d2d3ddc34905bf4ca9cc877d4e86 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:13:57.318333 27985 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f71d2d3ddc34905bf4ca9cc877d4e86 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:13:57.318432 27985 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f71d2d3ddc34905bf4ca9cc877d4e86 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:13:57.319564 27985 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f71d2d3ddc34905bf4ca9cc877d4e86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f71d2d3ddc34905bf4ca9cc877d4e86" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 37887 } }
I20260620 19:13:57.319948 27985 leader_election.cc:302] T 00000000000000000000000000000000 P 8f71d2d3ddc34905bf4ca9cc877d4e86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f71d2d3ddc34905bf4ca9cc877d4e86; no voters:
I20260620 19:13:57.320320 27985 leader_election.cc:288] T 00000000000000000000000000000000 P 8f71d2d3ddc34905bf4ca9cc877d4e86 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:13:57.320389 27990 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f71d2d3ddc34905bf4ca9cc877d4e86 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:13:57.320590 27990 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f71d2d3ddc34905bf4ca9cc877d4e86 [term 1 LEADER]: Becoming Leader. State: Replica: 8f71d2d3ddc34905bf4ca9cc877d4e86, State: Running, Role: LEADER
I20260620 19:13:57.320883 27990 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f71d2d3ddc34905bf4ca9cc877d4e86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f71d2d3ddc34905bf4ca9cc877d4e86" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 37887 } }
I20260620 19:13:57.321645 27985 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f71d2d3ddc34905bf4ca9cc877d4e86 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:13:57.323647 27988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:57.309503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60862 (local address 127.9.155.126:37887)
0620 19:13:57.309896 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:57.309915 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:57.309957 (+ 42us) server_negotiation.cc:407] Connection header received
0620 19:13:57.310650 (+ 693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:57.310669 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:57.311045 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:57.311431 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:57.312344 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:57.313391 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:57.314601 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:57.315094 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:57.315223 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:57.318040 (+ 2817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:13:57.318045 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:13:57.318060 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:13:57.318090 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:13:57.320060 (+ 1970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:57.320570 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:57.320578 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:57.320584 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:57.320743 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:57.321058 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:57.321062 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:57.321064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:57.321536 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:13:57.321702 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:57.322435 (+ 733us) server_negotiation.cc:299] Negotiation successful
0620 19:13:57.322732 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":109,"threads_started":1}
I20260620 19:13:57.324095 27991 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f71d2d3ddc34905bf4ca9cc877d4e86 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f71d2d3ddc34905bf4ca9cc877d4e86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f71d2d3ddc34905bf4ca9cc877d4e86" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 37887 } } }
I20260620 19:13:57.324771 27991 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f71d2d3ddc34905bf4ca9cc877d4e86 [sys.catalog]: This master's current role is: LEADER
I20260620 19:13:57.324302 27992 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f71d2d3ddc34905bf4ca9cc877d4e86 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f71d2d3ddc34905bf4ca9cc877d4e86. Latest consensus state: current_term: 1 leader_uuid: "8f71d2d3ddc34905bf4ca9cc877d4e86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f71d2d3ddc34905bf4ca9cc877d4e86" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 37887 } } }
I20260620 19:13:57.324942 27992 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f71d2d3ddc34905bf4ca9cc877d4e86 [sys.catalog]: This master's current role is: LEADER
W20260620 19:13:57.326602 28005 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8f71d2d3ddc34905bf4ca9cc877d4e86: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:13:57.326670 28005 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:13:57.326925 28006 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:13:57.327622 28006 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:13:57.332768 28006 catalog_manager.cc:1347] Generated new cluster ID: de7236e2887b49f5aa9f66166f611ee8
I20260620 19:13:57.332849 28006 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:13:57.344904 28006 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:13:57.345846 28006 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:13:57.362210 28006 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f71d2d3ddc34905bf4ca9cc877d4e86: Generated new TSK 0
I20260620 19:13:57.362924 28006 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.155.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.155.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.155.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.65.keytab.
Entry for principal kudu/127.9.155.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.65.keytab.
I20260620 19:13:57.394865 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.65:0
--local_ip_for_outbound_sockets=127.9.155.65
--webserver_interface=127.9.155.65
--webserver_port=0
--tserver_master_addrs=127.9.155.126:37887
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.65.keytab
--principal=kudu/127.9.155.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:37861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:13:57.528926 28011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:13:57.529210 28011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:13:57.529254 28011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:13:57.533537 28011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:13:57.533610 28011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:13:57.533671 28011 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.65
I20260620 19:13:57.539108 28011 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:37861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.65.keytab
--principal=kudu/127.9.155.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.155.65
--webserver_port=0
--tserver_master_addrs=127.9.155.126:37887
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28011
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.65
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:13:57.540196 28011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:13:57.541244 28011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:13:57.551034 28016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:13:57.551096 28017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:13:57.551076 28019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:13:57.551151 28011 server_base.cc:1034] running on GCE node
I20260620 19:13:57.551648 28011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:13:57.552291 28011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:13:57.553457 28011 hybrid_clock.cc:648] HybridClock initialized: now 1781982837553436 us; error 35 us; skew 500 ppm
Jun 20 19:13:57 dist-test-slave-n3fx krb5kdc[27904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982837, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:13:57.556300 28011 init.cc:377] Logged in from keytab as kudu/127.9.155.65@KRBTEST.COM (short username kudu)
I20260620 19:13:57.557188 28011 webserver.cc:460] Webserver started at http://127.9.155.65:36943/ using document root <none> and password file <none>
I20260620 19:13:57.557727 28011 fs_manager.cc:359] Metadata directory not provided
I20260620 19:13:57.557794 28011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:13:57.557987 28011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:13:57.560021 28011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/minicluster-data/ts-0/data/instance:
uuid: "577f4e9d83434b5f8fab117d80947d3a"
format_stamp: "Formatted at 2026-06-20 19:13:57 on dist-test-slave-n3fx"
server_key: "cd6cb2d4a6bc8c9d40942d9a33583eb1"
server_key_iv: "c24fb2fd1439cc100e4a54304cb33bec"
server_key_version: "encryptionkey@0"
I20260620 19:13:57.560581 28011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/minicluster-data/ts-0/wal/instance:
uuid: "577f4e9d83434b5f8fab117d80947d3a"
format_stamp: "Formatted at 2026-06-20 19:13:57 on dist-test-slave-n3fx"
server_key: "cd6cb2d4a6bc8c9d40942d9a33583eb1"
server_key_iv: "c24fb2fd1439cc100e4a54304cb33bec"
server_key_version: "encryptionkey@0"
I20260620 19:13:57.564255 28011 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:13:57.566732 28026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:57.567961 28011 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 19:13:57.568135 28011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/minicluster-data/ts-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/minicluster-data/ts-0/wal
uuid: "577f4e9d83434b5f8fab117d80947d3a"
format_stamp: "Formatted at 2026-06-20 19:13:57 on dist-test-slave-n3fx"
server_key: "cd6cb2d4a6bc8c9d40942d9a33583eb1"
server_key_iv: "c24fb2fd1439cc100e4a54304cb33bec"
server_key_version: "encryptionkey@0"
I20260620 19:13:57.568277 28011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:13:57.591794 28011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:13:57.594844 28011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:13:57.595036 28011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:13:57.595608 28011 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:13:57.596465 28011 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:13:57.596521 28011 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:57.596575 28011 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:13:57.596601 28011 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:57.605064 28011 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.65:46619
I20260620 19:13:57.605142 28139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.65:46619 every 8 connection(s)
I20260620 19:13:57.605999 28011 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
I20260620 19:13:57.612078 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 28011
I20260620 19:13:57.612236 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/minicluster-data/ts-0/wal/instance
I20260620 19:13:57.612498 9837 external_mini_cluster.cc:1428] Setting key e74698fe8c96a6b76abe07b01972149b
Jun 20 19:13:57 dist-test-slave-n3fx krb5kdc[27904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982837, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.65@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:13:57.619555 27988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:57.607812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.65:43177 (local address 127.9.155.126:37887)
0620 19:13:57.607955 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:57.607959 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:57.609082 (+ 1123us) server_negotiation.cc:407] Connection header received
0620 19:13:57.609942 (+ 860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:57.609946 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:57.609997 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:57.610079 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:57.611296 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:57.611843 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:57.613205 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:57.613440 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:57.613529 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:57.615937 (+ 2408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:13:57.615940 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:13:57.615942 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:13:57.615968 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:13:57.617364 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:57.617940 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:57.617942 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:57.617943 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:57.618013 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:57.618361 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:57.618363 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:57.618364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:57.618561 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:13:57.618637 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:57.619297 (+ 660us) server_negotiation.cc:299] Negotiation successful
0620 19:13:57.619412 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 19:13:57.620736 28142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:57.608210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:37887 (local address 127.9.155.65:43177)
0620 19:13:57.608910 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:57.608954 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:57.609730 (+ 776us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:57.610274 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:57.610281 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:57.610671 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:57.611140 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:57.611154 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:57.612026 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:57.612029 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:57.613096 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:57.613106 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:57.613599 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:57.613603 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:57.613797 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:57.614400 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:57.614417 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:57.615763 (+ 1346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:13:57.617471 (+ 1708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:57.617486 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:57.617503 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:57.617851 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:57.618091 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:57.618094 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:57.618096 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:57.618279 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:57.618659 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:13:57.618666 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:13:57.619021 (+ 355us) client_negotiation.cc:769] Sending connection context
0620 19:13:57.619250 (+ 229us) client_negotiation.cc:240] Negotiation successful
0620 19:13:57.619620 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":112,"threads_started":1}
WARNING: no policy specified for kudu/127.9.155.66@KRBTEST.COM; defaulting to no policy
I20260620 19:13:57.622085 28140 heartbeater.cc:347] Connected to a master server at 127.9.155.126:37887
I20260620 19:13:57.622298 28140 heartbeater.cc:464] Registering TS with master...
I20260620 19:13:57.622846 28140 heartbeater.cc:511] Master 127.9.155.126:37887 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.155.66@KRBTEST.COM" created.
I20260620 19:13:57.624492 27949 ts_manager.cc:194] Registered new tserver with Master: 577f4e9d83434b5f8fab117d80947d3a (127.9.155.65:46619)
I20260620 19:13:57.625650 27949 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.155.65@KRBTEST.COM'} at 127.9.155.65:43177
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.155.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.66.keytab.
Entry for principal kudu/127.9.155.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.66.keytab.
I20260620 19:13:57.639392 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.66:0
--local_ip_for_outbound_sockets=127.9.155.66
--webserver_interface=127.9.155.66
--webserver_port=0
--tserver_master_addrs=127.9.155.126:37887
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.66.keytab
--principal=kudu/127.9.155.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:37861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:13:57.748443 28145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:13:57.748705 28145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:13:57.748765 28145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:13:57.752795 28145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:13:57.752857 28145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:13:57.752939 28145 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.66
I20260620 19:13:57.758237 28145 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:37861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.66.keytab
--principal=kudu/127.9.155.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.155.66
--webserver_port=0
--tserver_master_addrs=127.9.155.126:37887
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.66
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:13:57.759315 28145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:13:57.760358 28145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:13:57.770290 28151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:13:57.770290 28150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:13:57.770298 28153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:13:57.770682 28145 server_base.cc:1034] running on GCE node
I20260620 19:13:57.771118 28145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:13:57.771849 28145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:13:57.773080 28145 hybrid_clock.cc:648] HybridClock initialized: now 1781982837773058 us; error 36 us; skew 500 ppm
Jun 20 19:13:57 dist-test-slave-n3fx krb5kdc[27904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982837, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:13:57.779035 28145 init.cc:377] Logged in from keytab as kudu/127.9.155.66@KRBTEST.COM (short username kudu)
I20260620 19:13:57.780032 28145 webserver.cc:460] Webserver started at http://127.9.155.66:41915/ using document root <none> and password file <none>
I20260620 19:13:57.780637 28145 fs_manager.cc:359] Metadata directory not provided
I20260620 19:13:57.780704 28145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:13:57.780891 28145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:13:57.783002 28145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/minicluster-data/ts-1/data/instance:
uuid: "03f81750d8c24cfbabdd558e02bef559"
format_stamp: "Formatted at 2026-06-20 19:13:57 on dist-test-slave-n3fx"
server_key: "81a386c594c6c3f6b11354ba09f5c7b9"
server_key_iv: "0c64195d0516fbba9f3f32e698704650"
server_key_version: "encryptionkey@0"
I20260620 19:13:57.783547 28145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/minicluster-data/ts-1/wal/instance:
uuid: "03f81750d8c24cfbabdd558e02bef559"
format_stamp: "Formatted at 2026-06-20 19:13:57 on dist-test-slave-n3fx"
server_key: "81a386c594c6c3f6b11354ba09f5c7b9"
server_key_iv: "0c64195d0516fbba9f3f32e698704650"
server_key_version: "encryptionkey@0"
I20260620 19:13:57.787320 28145 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 19:13:57.789947 28160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:57.791144 28145 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 19:13:57.791250 28145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/minicluster-data/ts-1/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/minicluster-data/ts-1/wal
uuid: "03f81750d8c24cfbabdd558e02bef559"
format_stamp: "Formatted at 2026-06-20 19:13:57 on dist-test-slave-n3fx"
server_key: "81a386c594c6c3f6b11354ba09f5c7b9"
server_key_iv: "0c64195d0516fbba9f3f32e698704650"
server_key_version: "encryptionkey@0"
I20260620 19:13:57.791363 28145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:13:57.811622 28145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:13:57.815042 28145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:13:57.815237 28145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:13:57.815867 28145 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:13:57.816777 28145 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:13:57.816829 28145 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:57.816884 28145 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:13:57.816910 28145 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:57.825722 28145 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.66:34831
I20260620 19:13:57.825752 28273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.66:34831 every 8 connection(s)
I20260620 19:13:57.826617 28145 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
Jun 20 19:13:57 dist-test-slave-n3fx krb5kdc[27904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982837, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.66@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:13:57.835759 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 28145
I20260620 19:13:57.835875 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/minicluster-data/ts-1/wal/instance
I20260620 19:13:57.836199 9837 external_mini_cluster.cc:1428] Setting key ab89acefbeece9dc9b397e9023dfed93
I20260620 19:13:57.839847 27988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:57.828543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.66:42475 (local address 127.9.155.126:37887)
0620 19:13:57.828673 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:57.828677 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:57.829521 (+ 844us) server_negotiation.cc:407] Connection header received
0620 19:13:57.830396 (+ 875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:57.830399 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:57.830454 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:57.830526 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:57.831687 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:57.832209 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:57.833319 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:57.833549 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:57.833646 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:57.836007 (+ 2361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:13:57.836011 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:13:57.836014 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:13:57.836047 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:13:57.837765 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:57.838308 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:57.838312 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:57.838314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:57.838396 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:57.838724 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:57.838727 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:57.838729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:57.838913 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:13:57.839000 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:57.839614 (+ 614us) server_negotiation.cc:299] Negotiation successful
0620 19:13:57.839724 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 19:13:57.841034 28276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:57.828919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:37887 (local address 127.9.155.66:42475)
0620 19:13:57.829357 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:57.829400 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:57.830204 (+ 804us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:57.830695 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:57.830702 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:57.831071 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:57.831555 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:57.831569 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:57.832339 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:57.832341 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:57.833203 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:57.833212 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:57.833690 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:57.833694 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:57.833852 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:57.834393 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:57.834408 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:57.835811 (+ 1403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:13:57.837889 (+ 2078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:57.837896 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:57.837915 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:57.838212 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:57.838479 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:57.838483 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:57.838485 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:57.838636 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:57.839014 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:13:57.839022 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:13:57.839308 (+ 286us) client_negotiation.cc:769] Sending connection context
0620 19:13:57.839553 (+ 245us) client_negotiation.cc:240] Negotiation successful
0620 19:13:57.839929 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":112,"threads_started":1}
I20260620 19:13:57.842293 28274 heartbeater.cc:347] Connected to a master server at 127.9.155.126:37887
I20260620 19:13:57.842541 28274 heartbeater.cc:464] Registering TS with master...
I20260620 19:13:57.843048 28274 heartbeater.cc:511] Master 127.9.155.126:37887 requested a full tablet report, sending...
I20260620 19:13:57.844254 27949 ts_manager.cc:194] Registered new tserver with Master: 03f81750d8c24cfbabdd558e02bef559 (127.9.155.66:34831)
I20260620 19:13:57.844838 27949 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.155.66@KRBTEST.COM'} at 127.9.155.66:42475
WARNING: no policy specified for kudu/127.9.155.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.155.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.155.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.67.keytab.
Entry for principal kudu/127.9.155.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.67.keytab.
I20260620 19:13:57.867914 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.67:0
--local_ip_for_outbound_sockets=127.9.155.67
--webserver_interface=127.9.155.67
--webserver_port=0
--tserver_master_addrs=127.9.155.126:37887
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.67.keytab
--principal=kudu/127.9.155.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:37861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:13:57.977546 28279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:13:57.977811 28279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:13:57.977871 28279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:13:57.981954 28279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:13:57.982040 28279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:13:57.982121 28279 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.67
I20260620 19:13:57.987612 28279 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:37861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.67.keytab
--principal=kudu/127.9.155.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.155.67
--webserver_port=0
--tserver_master_addrs=127.9.155.126:37887
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28279
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.67
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:13:57.988718 28279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:13:57.989722 28279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:13:57.999429 28284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:13:57.999428 28285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:13:57.999711 28279 server_base.cc:1034] running on GCE node
W20260620 19:13:57.999938 28287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:13:58.000394 28279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:13:58.001056 28279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:13:58.002241 28279 hybrid_clock.cc:648] HybridClock initialized: now 1781982838002222 us; error 33 us; skew 500 ppm
Jun 20 19:13:58 dist-test-slave-n3fx krb5kdc[27904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982838, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:13:58.005254 28279 init.cc:377] Logged in from keytab as kudu/127.9.155.67@KRBTEST.COM (short username kudu)
I20260620 19:13:58.006255 28279 webserver.cc:460] Webserver started at http://127.9.155.67:36713/ using document root <none> and password file <none>
I20260620 19:13:58.006844 28279 fs_manager.cc:359] Metadata directory not provided
I20260620 19:13:58.006951 28279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:13:58.007171 28279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:13:58.009333 28279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/minicluster-data/ts-2/data/instance:
uuid: "4556a2448ba3483a89a4dad9fd1dcab3"
format_stamp: "Formatted at 2026-06-20 19:13:58 on dist-test-slave-n3fx"
server_key: "30426352b63b05ff73089792a620f7fd"
server_key_iv: "f66d393c4bc43ea55ffd203beabfcc0d"
server_key_version: "encryptionkey@0"
I20260620 19:13:58.009917 28279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/minicluster-data/ts-2/wal/instance:
uuid: "4556a2448ba3483a89a4dad9fd1dcab3"
format_stamp: "Formatted at 2026-06-20 19:13:58 on dist-test-slave-n3fx"
server_key: "30426352b63b05ff73089792a620f7fd"
server_key_iv: "f66d393c4bc43ea55ffd203beabfcc0d"
server_key_version: "encryptionkey@0"
I20260620 19:13:58.013782 28279 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 19:13:58.016602 28294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:58.018235 28279 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 19:13:58.018378 28279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/minicluster-data/ts-2/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/minicluster-data/ts-2/wal
uuid: "4556a2448ba3483a89a4dad9fd1dcab3"
format_stamp: "Formatted at 2026-06-20 19:13:58 on dist-test-slave-n3fx"
server_key: "30426352b63b05ff73089792a620f7fd"
server_key_iv: "f66d393c4bc43ea55ffd203beabfcc0d"
server_key_version: "encryptionkey@0"
I20260620 19:13:58.018515 28279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:13:58.036686 28279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:13:58.039724 28279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:13:58.039901 28279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:13:58.040518 28279 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:13:58.041332 28279 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:13:58.041378 28279 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:58.041429 28279 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:13:58.041457 28279 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:58.049952 28279 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.67:40157
I20260620 19:13:58.049984 28407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.67:40157 every 8 connection(s)
I20260620 19:13:58.050813 28279 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
I20260620 19:13:58.054289 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 28279
I20260620 19:13:58.054415 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/minicluster-data/ts-2/wal/instance
I20260620 19:13:58.054685 9837 external_mini_cluster.cc:1428] Setting key 1a6849789c112fd55922bdb88c0addd7
Jun 20 19:13:58 dist-test-slave-n3fx krb5kdc[27904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982838, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.67@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:13:58.064671 27988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:58.052592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.67:51357 (local address 127.9.155.126:37887)
0620 19:13:58.052724 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:58.052728 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:58.053576 (+ 848us) server_negotiation.cc:407] Connection header received
0620 19:13:58.054510 (+ 934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:58.054514 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:58.054572 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:58.054619 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:58.056316 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:58.056943 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:58.058129 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:58.058352 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:58.058445 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:58.061124 (+ 2679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:13:58.061127 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:13:58.061130 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:13:58.061157 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:13:58.062510 (+ 1353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:58.063047 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:58.063050 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:58.063051 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:58.063139 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:58.063544 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:58.063547 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:58.063548 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:58.063753 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:13:58.063843 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:58.064423 (+ 580us) server_negotiation.cc:299] Negotiation successful
0620 19:13:58.064545 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 19:13:58.065469 28410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:58.052985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:37887 (local address 127.9.155.67:51357)
0620 19:13:58.053419 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:58.053461 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:58.054280 (+ 819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:58.054969 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:58.054978 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:58.055463 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:58.056112 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:58.056137 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:58.057068 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:58.057078 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:58.058003 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:58.058013 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:58.058508 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:58.058512 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:58.058679 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:58.059259 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:58.059276 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:58.060899 (+ 1623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:13:58.062631 (+ 1732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:58.062637 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:58.062651 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:58.062913 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:58.063252 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:58.063255 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:58.063257 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:58.063417 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:58.063873 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:13:58.063878 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:13:58.064177 (+ 299us) client_negotiation.cc:769] Sending connection context
0620 19:13:58.064363 (+ 186us) client_negotiation.cc:240] Negotiation successful
0620 19:13:58.064639 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":111,"threads_started":1}
I20260620 19:13:58.066723 28408 heartbeater.cc:347] Connected to a master server at 127.9.155.126:37887
I20260620 19:13:58.067019 28408 heartbeater.cc:464] Registering TS with master...
I20260620 19:13:58.067561 28408 heartbeater.cc:511] Master 127.9.155.126:37887 requested a full tablet report, sending...
I20260620 19:13:58.068642 27948 ts_manager.cc:194] Registered new tserver with Master: 4556a2448ba3483a89a4dad9fd1dcab3 (127.9.155.67:40157)
I20260620 19:13:58.069228 27948 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.155.67@KRBTEST.COM'} at 127.9.155.67:51357
I20260620 19:13:58.080860 9837 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:13:58.090364 27988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:58.082837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37324 (local address 127.9.155.126:37887)
0620 19:13:58.082970 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:58.082973 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:58.083135 (+ 162us) server_negotiation.cc:407] Connection header received
0620 19:13:58.083226 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:58.083229 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:58.083273 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:58.083335 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:58.084033 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:58.084684 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:58.085534 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:58.085760 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:58.085851 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:58.087360 (+ 1509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:13:58.087363 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:13:58.087365 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:13:58.087391 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:13:58.088682 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:58.089171 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:58.089174 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:58.089175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:58.089255 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:58.089558 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:58.089561 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:58.089562 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:58.089784 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:13:58.089901 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:58.090144 (+ 243us) server_negotiation.cc:299] Negotiation successful
0620 19:13:58.090239 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 19:13:58.093520 27948 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37324:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:13:58.096571 27948 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:13:58.110544 28422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:58.104616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.65:46619 (local address 127.0.0.1:36882)
0620 19:13:58.105495 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:58.105508 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:58.105591 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:58.106452 (+ 861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:58.106455 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:58.106472 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:13:58.106554 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:58.106559 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:58.107847 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:58.107851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:58.108894 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:58.108903 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:58.109985 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:58.109989 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:58.110103 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:58.110146 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 19:13:58.110258 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 19:13:58.110382 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":822,"mutex_wait_us":24,"thread_start_us":94,"threads_started":1}
I20260620 19:13:58.110699 28424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:58.105680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36882 (local address 127.9.155.65:46619)
0620 19:13:58.106047 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:58.106060 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:58.106080 (+ 20us) server_negotiation.cc:407] Connection header received
0620 19:13:58.106143 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:58.106148 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:58.106332 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:58.106490 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:13:58.106639 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:58.107729 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:58.109054 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:58.109864 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:58.109991 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:58.110150 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:58.110462 (+ 312us) server_negotiation.cc:299] Negotiation successful
0620 19:13:58.110563 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":122,"threads_started":1}
I20260620 19:13:58.110831 28421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:58.104691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36868 (local address 127.9.155.67:40157)
0620 19:13:58.105019 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:58.105031 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:58.105327 (+ 296us) server_negotiation.cc:407] Connection header received
0620 19:13:58.105575 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:58.105582 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:58.105782 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:58.105918 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:13:58.106355 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:58.107554 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:58.109351 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:58.110130 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:58.110231 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:58.110363 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:58.110600 (+ 237us) server_negotiation.cc:299] Negotiation successful
0620 19:13:58.110683 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":131,"threads_started":1}
I20260620 19:13:58.111058 28419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:58.104553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.67:40157 (local address 127.0.0.1:36868)
0620 19:13:58.105196 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:58.105245 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:58.105382 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:58.106104 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:58.106109 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:58.106130 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:13:58.106252 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:58.106258 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:58.107691 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:58.107703 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:58.109246 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:58.109257 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:58.110231 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:58.110235 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:58.110322 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:58.110344 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 19:13:58.110429 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 19:13:58.110502 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"mutex_wait_us":176,"thread_start_us":86,"threads_started":1}
I20260620 19:13:58.111917 28420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:58.105289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.66:34831 (local address 127.0.0.1:33632)
0620 19:13:58.106508 (+ 1219us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:58.106519 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:58.106598 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:58.107182 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:58.107184 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:58.107198 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:13:58.107260 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:58.107264 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:58.108531 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:58.108535 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:58.109545 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:58.109553 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:58.110618 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:58.110621 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:58.110724 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:58.110745 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 19:13:58.110807 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 19:13:58.110866 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1165,"mutex_wait_us":2,"thread_start_us":88,"threads_started":1}
I20260620 19:13:58.111225 28423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:58.105454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33632 (local address 127.9.155.66:34831)
0620 19:13:58.106015 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:58.106025 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:58.106684 (+ 659us) server_negotiation.cc:407] Connection header received
0620 19:13:58.106754 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:58.106758 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:58.106888 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:58.107034 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:13:58.107364 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:58.108387 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:58.109657 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:58.110348 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:58.110437 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:58.110557 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:58.110983 (+ 426us) server_negotiation.cc:299] Negotiation successful
0620 19:13:58.111086 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":102,"threads_started":1}
I20260620 19:13:58.113896 28342 tablet_service.cc:1467] Processing CreateTablet for tablet 4bb6760c106c4253a4f90112d6a68528 (DEFAULT_TABLE table=test-table [id=cc50698e92f94d0c828f7afd39e030f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:13:58.113924 28074 tablet_service.cc:1467] Processing CreateTablet for tablet 4bb6760c106c4253a4f90112d6a68528 (DEFAULT_TABLE table=test-table [id=cc50698e92f94d0c828f7afd39e030f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:13:58.114909 28342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bb6760c106c4253a4f90112d6a68528. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:13:58.114948 28074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bb6760c106c4253a4f90112d6a68528. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:13:58.115183 28208 tablet_service.cc:1467] Processing CreateTablet for tablet 4bb6760c106c4253a4f90112d6a68528 (DEFAULT_TABLE table=test-table [id=cc50698e92f94d0c828f7afd39e030f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:13:58.116183 28208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bb6760c106c4253a4f90112d6a68528. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:13:58.120810 28425 tablet_bootstrap.cc:492] T 4bb6760c106c4253a4f90112d6a68528 P 4556a2448ba3483a89a4dad9fd1dcab3: Bootstrap starting.
I20260620 19:13:58.122298 28426 tablet_bootstrap.cc:492] T 4bb6760c106c4253a4f90112d6a68528 P 577f4e9d83434b5f8fab117d80947d3a: Bootstrap starting.
I20260620 19:13:58.123449 28427 tablet_bootstrap.cc:492] T 4bb6760c106c4253a4f90112d6a68528 P 03f81750d8c24cfbabdd558e02bef559: Bootstrap starting.
I20260620 19:13:58.123618 28425 tablet_bootstrap.cc:654] T 4bb6760c106c4253a4f90112d6a68528 P 4556a2448ba3483a89a4dad9fd1dcab3: Neither blocks nor log segments found. Creating new log.
I20260620 19:13:58.124677 28426 tablet_bootstrap.cc:654] T 4bb6760c106c4253a4f90112d6a68528 P 577f4e9d83434b5f8fab117d80947d3a: Neither blocks nor log segments found. Creating new log.
I20260620 19:13:58.125108 28425 log.cc:824] T 4bb6760c106c4253a4f90112d6a68528 P 4556a2448ba3483a89a4dad9fd1dcab3: Log is configured to *not* fsync() on all Append() calls
I20260620 19:13:58.125473 28426 log.cc:824] T 4bb6760c106c4253a4f90112d6a68528 P 577f4e9d83434b5f8fab117d80947d3a: Log is configured to *not* fsync() on all Append() calls
I20260620 19:13:58.125625 28427 tablet_bootstrap.cc:654] T 4bb6760c106c4253a4f90112d6a68528 P 03f81750d8c24cfbabdd558e02bef559: Neither blocks nor log segments found. Creating new log.
I20260620 19:13:58.126421 28427 log.cc:824] T 4bb6760c106c4253a4f90112d6a68528 P 03f81750d8c24cfbabdd558e02bef559: Log is configured to *not* fsync() on all Append() calls
I20260620 19:13:58.127101 28425 tablet_bootstrap.cc:492] T 4bb6760c106c4253a4f90112d6a68528 P 4556a2448ba3483a89a4dad9fd1dcab3: No bootstrap required, opened a new log
I20260620 19:13:58.127293 28425 ts_tablet_manager.cc:1397] T 4bb6760c106c4253a4f90112d6a68528 P 4556a2448ba3483a89a4dad9fd1dcab3: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260620 19:13:58.127322 28426 tablet_bootstrap.cc:492] T 4bb6760c106c4253a4f90112d6a68528 P 577f4e9d83434b5f8fab117d80947d3a: No bootstrap required, opened a new log
I20260620 19:13:58.127485 28426 ts_tablet_manager.cc:1397] T 4bb6760c106c4253a4f90112d6a68528 P 577f4e9d83434b5f8fab117d80947d3a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 19:13:58.128283 28427 tablet_bootstrap.cc:492] T 4bb6760c106c4253a4f90112d6a68528 P 03f81750d8c24cfbabdd558e02bef559: No bootstrap required, opened a new log
I20260620 19:13:58.128463 28427 ts_tablet_manager.cc:1397] T 4bb6760c106c4253a4f90112d6a68528 P 03f81750d8c24cfbabdd558e02bef559: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260620 19:13:58.130293 28425 raft_consensus.cc:348] T 4bb6760c106c4253a4f90112d6a68528 P 4556a2448ba3483a89a4dad9fd1dcab3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4556a2448ba3483a89a4dad9fd1dcab3" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 40157 } } peers { permanent_uuid: "577f4e9d83434b5f8fab117d80947d3a" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 46619 } } peers { permanent_uuid: "03f81750d8c24cfbabdd558e02bef559" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 34831 } }
I20260620 19:13:58.130512 28425 raft_consensus.cc:374] T 4bb6760c106c4253a4f90112d6a68528 P 4556a2448ba3483a89a4dad9fd1dcab3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:13:58.130563 28425 raft_consensus.cc:729] T 4bb6760c106c4253a4f90112d6a68528 P 4556a2448ba3483a89a4dad9fd1dcab3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4556a2448ba3483a89a4dad9fd1dcab3, State: Initialized, Role: FOLLOWER
I20260620 19:13:58.130978 28425 consensus_queue.cc:260] T 4bb6760c106c4253a4f90112d6a68528 P 4556a2448ba3483a89a4dad9fd1dcab3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4556a2448ba3483a89a4dad9fd1dcab3" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 40157 } } peers { permanent_uuid: "577f4e9d83434b5f8fab117d80947d3a" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 46619 } } peers { permanent_uuid: "03f81750d8c24cfbabdd558e02bef559" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 34831 } }
I20260620 19:13:58.131938 28425 ts_tablet_manager.cc:1428] T 4bb6760c106c4253a4f90112d6a68528 P 4556a2448ba3483a89a4dad9fd1dcab3: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 19:13:58.131806 28426 raft_consensus.cc:348] T 4bb6760c106c4253a4f90112d6a68528 P 577f4e9d83434b5f8fab117d80947d3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4556a2448ba3483a89a4dad9fd1dcab3" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 40157 } } peers { permanent_uuid: "577f4e9d83434b5f8fab117d80947d3a" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 46619 } } peers { permanent_uuid: "03f81750d8c24cfbabdd558e02bef559" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 34831 } }
I20260620 19:13:58.132082 28426 raft_consensus.cc:374] T 4bb6760c106c4253a4f90112d6a68528 P 577f4e9d83434b5f8fab117d80947d3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:13:58.131966 28427 raft_consensus.cc:348] T 4bb6760c106c4253a4f90112d6a68528 P 03f81750d8c24cfbabdd558e02bef559 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4556a2448ba3483a89a4dad9fd1dcab3" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 40157 } } peers { permanent_uuid: "577f4e9d83434b5f8fab117d80947d3a" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 46619 } } peers { permanent_uuid: "03f81750d8c24cfbabdd558e02bef559" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 34831 } }
I20260620 19:13:58.132138 28426 raft_consensus.cc:729] T 4bb6760c106c4253a4f90112d6a68528 P 577f4e9d83434b5f8fab117d80947d3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 577f4e9d83434b5f8fab117d80947d3a, State: Initialized, Role: FOLLOWER
I20260620 19:13:58.132193 28427 raft_consensus.cc:374] T 4bb6760c106c4253a4f90112d6a68528 P 03f81750d8c24cfbabdd558e02bef559 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:13:58.132256 28427 raft_consensus.cc:729] T 4bb6760c106c4253a4f90112d6a68528 P 03f81750d8c24cfbabdd558e02bef559 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03f81750d8c24cfbabdd558e02bef559, State: Initialized, Role: FOLLOWER
I20260620 19:13:58.132306 28408 heartbeater.cc:503] Master 127.9.155.126:37887 was elected leader, sending a full tablet report...
I20260620 19:13:58.132592 28426 consensus_queue.cc:260] T 4bb6760c106c4253a4f90112d6a68528 P 577f4e9d83434b5f8fab117d80947d3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4556a2448ba3483a89a4dad9fd1dcab3" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 40157 } } peers { permanent_uuid: "577f4e9d83434b5f8fab117d80947d3a" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 46619 } } peers { permanent_uuid: "03f81750d8c24cfbabdd558e02bef559" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 34831 } }
I20260620 19:13:58.132627 28427 consensus_queue.cc:260] T 4bb6760c106c4253a4f90112d6a68528 P 03f81750d8c24cfbabdd558e02bef559 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4556a2448ba3483a89a4dad9fd1dcab3" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 40157 } } peers { permanent_uuid: "577f4e9d83434b5f8fab117d80947d3a" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 46619 } } peers { permanent_uuid: "03f81750d8c24cfbabdd558e02bef559" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 34831 } }
I20260620 19:13:58.133507 28426 ts_tablet_manager.cc:1428] T 4bb6760c106c4253a4f90112d6a68528 P 577f4e9d83434b5f8fab117d80947d3a: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 19:13:58.133507 28427 ts_tablet_manager.cc:1428] T 4bb6760c106c4253a4f90112d6a68528 P 03f81750d8c24cfbabdd558e02bef559: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 19:13:58.133575 28140 heartbeater.cc:503] Master 127.9.155.126:37887 was elected leader, sending a full tablet report...
I20260620 19:13:58.133740 28274 heartbeater.cc:503] Master 127.9.155.126:37887 was elected leader, sending a full tablet report...
I20260620 19:13:58.165347 28431 raft_consensus.cc:482] T 4bb6760c106c4253a4f90112d6a68528 P 4556a2448ba3483a89a4dad9fd1dcab3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:13:58.165549 28431 raft_consensus.cc:504] T 4bb6760c106c4253a4f90112d6a68528 P 4556a2448ba3483a89a4dad9fd1dcab3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4556a2448ba3483a89a4dad9fd1dcab3" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 40157 } } peers { permanent_uuid: "577f4e9d83434b5f8fab117d80947d3a" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 46619 } } peers { permanent_uuid: "03f81750d8c24cfbabdd558e02bef559" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 34831 } }
I20260620 19:13:58.166633 28431 leader_election.cc:288] T 4bb6760c106c4253a4f90112d6a68528 P 4556a2448ba3483a89a4dad9fd1dcab3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 577f4e9d83434b5f8fab117d80947d3a (127.9.155.65:46619), 03f81750d8c24cfbabdd558e02bef559 (127.9.155.66:34831)
I20260620 19:13:58.170271 28410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:58.166938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.65:46619 (local address 127.9.155.67:34739)
0620 19:13:58.167079 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:58.167095 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:58.167199 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:58.167500 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:58.167503 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:58.167521 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:13:58.167620 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:58.167625 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:58.168424 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:58.168428 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:58.169262 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:58.169271 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:58.169876 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:58.169880 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:58.169983 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:58.170002 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 19:13:58.170064 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 19:13:58.170135 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":23}
I20260620 19:13:58.170271 28424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:58.167022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.67:34739 (local address 127.9.155.65:46619)
0620 19:13:58.167170 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:58.167175 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:58.167192 (+ 17us) server_negotiation.cc:407] Connection header received
0620 19:13:58.167308 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:58.167311 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:58.167366 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:58.167447 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:13:58.167724 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:58.168317 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:58.169378 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:58.169771 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:58.169876 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:58.169929 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:58.170087 (+ 158us) server_negotiation.cc:299] Negotiation successful
0620 19:13:58.170153 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 19:13:58.170596 28423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:58.167328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.67:54967 (local address 127.9.155.66:34831)
0620 19:13:58.167445 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:58.167449 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:58.167643 (+ 194us) server_negotiation.cc:407] Connection header received
0620 19:13:58.167744 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:58.167747 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:58.167790 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:58.167854 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:13:58.168099 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:58.168636 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:58.169629 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:58.170068 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:58.170171 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:58.170218 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:58.170402 (+ 184us) server_negotiation.cc:299] Negotiation successful
0620 19:13:58.170488 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 19:13:58.170975 28434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:58.167192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.66:34831 (local address 127.9.155.67:54967)
0620 19:13:58.167565 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:58.167575 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:58.167663 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:58.167870 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:58.167873 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:58.167887 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:13:58.168006 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:58.168011 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:58.168732 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:58.168735 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:58.169518 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:58.169527 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:58.170167 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:58.170171 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:58.170267 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:58.170289 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 19:13:58.170778 (+ 489us) client_negotiation.cc:240] Negotiation successful
0620 19:13:58.170845 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"mutex_wait_us":1,"thread_start_us":129,"threads_started":1}
I20260620 19:13:58.170970 28094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bb6760c106c4253a4f90112d6a68528" candidate_uuid: "4556a2448ba3483a89a4dad9fd1dcab3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "577f4e9d83434b5f8fab117d80947d3a" is_pre_election: true
I20260620 19:13:58.171288 28094 raft_consensus.cc:2454] T 4bb6760c106c4253a4f90112d6a68528 P 577f4e9d83434b5f8fab117d80947d3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4556a2448ba3483a89a4dad9fd1dcab3 in term 0.
I20260620 19:13:58.171618 28228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bb6760c106c4253a4f90112d6a68528" candidate_uuid: "4556a2448ba3483a89a4dad9fd1dcab3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03f81750d8c24cfbabdd558e02bef559" is_pre_election: true
I20260620 19:13:58.171847 28297 leader_election.cc:302] T 4bb6760c106c4253a4f90112d6a68528 P 4556a2448ba3483a89a4dad9fd1dcab3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4556a2448ba3483a89a4dad9fd1dcab3, 577f4e9d83434b5f8fab117d80947d3a; no voters:
I20260620 19:13:58.171911 28228 raft_consensus.cc:2454] T 4bb6760c106c4253a4f90112d6a68528 P 03f81750d8c24cfbabdd558e02bef559 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4556a2448ba3483a89a4dad9fd1dcab3 in term 0.
I20260620 19:13:58.172161 28431 raft_consensus.cc:2781] T 4bb6760c106c4253a4f90112d6a68528 P 4556a2448ba3483a89a4dad9fd1dcab3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:13:58.172264 28431 raft_consensus.cc:482] T 4bb6760c106c4253a4f90112d6a68528 P 4556a2448ba3483a89a4dad9fd1dcab3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:13:58.172305 28431 raft_consensus.cc:3037] T 4bb6760c106c4253a4f90112d6a68528 P 4556a2448ba3483a89a4dad9fd1dcab3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:13:58.173328 28431 raft_consensus.cc:504] T 4bb6760c106c4253a4f90112d6a68528 P 4556a2448ba3483a89a4dad9fd1dcab3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4556a2448ba3483a89a4dad9fd1dcab3" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 40157 } } peers { permanent_uuid: "577f4e9d83434b5f8fab117d80947d3a" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 46619 } } peers { permanent_uuid: "03f81750d8c24cfbabdd558e02bef559" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 34831 } }
I20260620 19:13:58.173607 28431 leader_election.cc:288] T 4bb6760c106c4253a4f90112d6a68528 P 4556a2448ba3483a89a4dad9fd1dcab3 [CANDIDATE]: Term 1 election: Requested vote from peers 577f4e9d83434b5f8fab117d80947d3a (127.9.155.65:46619), 03f81750d8c24cfbabdd558e02bef559 (127.9.155.66:34831)
I20260620 19:13:58.173980 28228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bb6760c106c4253a4f90112d6a68528" candidate_uuid: "4556a2448ba3483a89a4dad9fd1dcab3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03f81750d8c24cfbabdd558e02bef559"
I20260620 19:13:58.173982 28094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bb6760c106c4253a4f90112d6a68528" candidate_uuid: "4556a2448ba3483a89a4dad9fd1dcab3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "577f4e9d83434b5f8fab117d80947d3a"
I20260620 19:13:58.174082 28094 raft_consensus.cc:3037] T 4bb6760c106c4253a4f90112d6a68528 P 577f4e9d83434b5f8fab117d80947d3a [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:13:58.174086 28228 raft_consensus.cc:3037] T 4bb6760c106c4253a4f90112d6a68528 P 03f81750d8c24cfbabdd558e02bef559 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:13:58.175074 28228 raft_consensus.cc:2454] T 4bb6760c106c4253a4f90112d6a68528 P 03f81750d8c24cfbabdd558e02bef559 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4556a2448ba3483a89a4dad9fd1dcab3 in term 1.
I20260620 19:13:58.175163 28094 raft_consensus.cc:2454] T 4bb6760c106c4253a4f90112d6a68528 P 577f4e9d83434b5f8fab117d80947d3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4556a2448ba3483a89a4dad9fd1dcab3 in term 1.
I20260620 19:13:58.175359 28297 leader_election.cc:302] T 4bb6760c106c4253a4f90112d6a68528 P 4556a2448ba3483a89a4dad9fd1dcab3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03f81750d8c24cfbabdd558e02bef559, 4556a2448ba3483a89a4dad9fd1dcab3; no voters:
I20260620 19:13:58.175510 28431 raft_consensus.cc:2781] T 4bb6760c106c4253a4f90112d6a68528 P 4556a2448ba3483a89a4dad9fd1dcab3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:13:58.175729 28431 raft_consensus.cc:686] T 4bb6760c106c4253a4f90112d6a68528 P 4556a2448ba3483a89a4dad9fd1dcab3 [term 1 LEADER]: Becoming Leader. State: Replica: 4556a2448ba3483a89a4dad9fd1dcab3, State: Running, Role: LEADER
I20260620 19:13:58.176043 28431 consensus_queue.cc:237] T 4bb6760c106c4253a4f90112d6a68528 P 4556a2448ba3483a89a4dad9fd1dcab3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4556a2448ba3483a89a4dad9fd1dcab3" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 40157 } } peers { permanent_uuid: "577f4e9d83434b5f8fab117d80947d3a" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 46619 } } peers { permanent_uuid: "03f81750d8c24cfbabdd558e02bef559" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 34831 } }
I20260620 19:13:58.179133 27947 catalog_manager.cc:5495] T 4bb6760c106c4253a4f90112d6a68528 P 4556a2448ba3483a89a4dad9fd1dcab3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4556a2448ba3483a89a4dad9fd1dcab3 (127.9.155.67). New cstate: current_term: 1 leader_uuid: "4556a2448ba3483a89a4dad9fd1dcab3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4556a2448ba3483a89a4dad9fd1dcab3" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 40157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "577f4e9d83434b5f8fab117d80947d3a" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 46619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03f81750d8c24cfbabdd558e02bef559" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 34831 } health_report { overall_health: UNKNOWN } } }
I20260620 19:13:58.195577 28421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:58.191870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36870 (local address 127.9.155.67:40157)
0620 19:13:58.192013 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:58.192018 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:58.192032 (+ 14us) server_negotiation.cc:407] Connection header received
0620 19:13:58.192204 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:58.192206 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:58.192314 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:58.192409 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:13:58.192644 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:58.193175 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:58.194054 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:58.194271 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:58.194358 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:58.194711 (+ 353us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:13:58.195118 (+ 407us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:13:58.195249 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:58.195367 (+ 118us) server_negotiation.cc:299] Negotiation successful
0620 19:13:58.195408 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 19:13:58.201409 28228 raft_consensus.cc:1261] T 4bb6760c106c4253a4f90112d6a68528 P 03f81750d8c24cfbabdd558e02bef559 [term 1 FOLLOWER]: Refusing update from remote peer 4556a2448ba3483a89a4dad9fd1dcab3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:13:58.201409 28094 raft_consensus.cc:1261] T 4bb6760c106c4253a4f90112d6a68528 P 577f4e9d83434b5f8fab117d80947d3a [term 1 FOLLOWER]: Refusing update from remote peer 4556a2448ba3483a89a4dad9fd1dcab3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:13:58.202165 28435 consensus_queue.cc:1035] T 4bb6760c106c4253a4f90112d6a68528 P 4556a2448ba3483a89a4dad9fd1dcab3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03f81750d8c24cfbabdd558e02bef559" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 34831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:13:58.202353 28431 consensus_queue.cc:1035] T 4bb6760c106c4253a4f90112d6a68528 P 4556a2448ba3483a89a4dad9fd1dcab3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "577f4e9d83434b5f8fab117d80947d3a" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 46619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:13:58.211001 28440 mvcc.cc:205] Tried to move back new op lower bound from 7299001705265487872 to 7299001705171791872. Current Snapshot: MvccSnapshot[applied={T|T < 7299001705265487872}]
I20260620 19:13:58.217988 27949 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:13:58.218151 27949 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:13:58.220690 27949 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8f71d2d3ddc34905bf4ca9cc877d4e86: Sending DeleteTablet for 3 replicas of tablet 4bb6760c106c4253a4f90112d6a68528
I20260620 19:13:58.221503 28342 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bb6760c106c4253a4f90112d6a68528 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:13:58 UTC) from {username='kudu', principal='kudu/127.9.155.126@KRBTEST.COM'} at 127.0.0.1:36868
I20260620 19:13:58.221779 28208 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bb6760c106c4253a4f90112d6a68528 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:13:58 UTC) from {username='kudu', principal='kudu/127.9.155.126@KRBTEST.COM'} at 127.0.0.1:33632
I20260620 19:13:58.221807 28074 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bb6760c106c4253a4f90112d6a68528 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:13:58 UTC) from {username='kudu', principal='kudu/127.9.155.126@KRBTEST.COM'} at 127.0.0.1:36882
I20260620 19:13:58.222328 28448 tablet_replica.cc:330] T 4bb6760c106c4253a4f90112d6a68528 P 577f4e9d83434b5f8fab117d80947d3a: stopping tablet replica
I20260620 19:13:58.222369 28449 tablet_replica.cc:330] T 4bb6760c106c4253a4f90112d6a68528 P 03f81750d8c24cfbabdd558e02bef559: stopping tablet replica
I20260620 19:13:58.222767 28449 raft_consensus.cc:2229] T 4bb6760c106c4253a4f90112d6a68528 P 03f81750d8c24cfbabdd558e02bef559 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:13:58.222769 28448 raft_consensus.cc:2229] T 4bb6760c106c4253a4f90112d6a68528 P 577f4e9d83434b5f8fab117d80947d3a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:13:58.222975 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 28011
I20260620 19:13:58.223201 28448 raft_consensus.cc:2258] T 4bb6760c106c4253a4f90112d6a68528 P 577f4e9d83434b5f8fab117d80947d3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:13:58.225765 28449 raft_consensus.cc:2258] T 4bb6760c106c4253a4f90112d6a68528 P 03f81750d8c24cfbabdd558e02bef559 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:13:58.226161 28447 tablet_replica.cc:330] T 4bb6760c106c4253a4f90112d6a68528 P 4556a2448ba3483a89a4dad9fd1dcab3: stopping tablet replica
I20260620 19:13:58.226476 28447 raft_consensus.cc:2229] T 4bb6760c106c4253a4f90112d6a68528 P 4556a2448ba3483a89a4dad9fd1dcab3 [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:13:58.226853 28447 raft_consensus.cc:2258] T 4bb6760c106c4253a4f90112d6a68528 P 4556a2448ba3483a89a4dad9fd1dcab3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:13:58.227609 28449 ts_tablet_manager.cc:1905] T 4bb6760c106c4253a4f90112d6a68528 P 03f81750d8c24cfbabdd558e02bef559: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:13:58.228268 28447 ts_tablet_manager.cc:1905] T 4bb6760c106c4253a4f90112d6a68528 P 4556a2448ba3483a89a4dad9fd1dcab3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 19:13:58.230408 27935 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.155.65:46619 (error 108)
I20260620 19:13:58.230593 28449 ts_tablet_manager.cc:1918] T 4bb6760c106c4253a4f90112d6a68528 P 03f81750d8c24cfbabdd558e02bef559: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:13:58.230762 28449 log.cc:1196] T 4bb6760c106c4253a4f90112d6a68528 P 03f81750d8c24cfbabdd558e02bef559: Deleting WAL directory at /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/minicluster-data/ts-1/wal/wals/4bb6760c106c4253a4f90112d6a68528
I20260620 19:13:58.231024 28447 ts_tablet_manager.cc:1918] T 4bb6760c106c4253a4f90112d6a68528 P 4556a2448ba3483a89a4dad9fd1dcab3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:13:58.231120 28447 log.cc:1196] T 4bb6760c106c4253a4f90112d6a68528 P 4556a2448ba3483a89a4dad9fd1dcab3: Deleting WAL directory at /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982595231230-9837-0/minicluster-data/ts-2/wal/wals/4bb6760c106c4253a4f90112d6a68528
I20260620 19:13:58.231247 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 28145
I20260620 19:13:58.231274 28449 ts_tablet_manager.cc:1939] T 4bb6760c106c4253a4f90112d6a68528 P 03f81750d8c24cfbabdd558e02bef559: Deleting consensus metadata
I20260620 19:13:58.231416 28447 ts_tablet_manager.cc:1939] T 4bb6760c106c4253a4f90112d6a68528 P 4556a2448ba3483a89a4dad9fd1dcab3: Deleting consensus metadata
I20260620 19:13:58.232894 27934 catalog_manager.cc:4841] TS 4556a2448ba3483a89a4dad9fd1dcab3 (127.9.155.67:40157): tablet 4bb6760c106c4253a4f90112d6a68528 (table test-table [id=cc50698e92f94d0c828f7afd39e030f3]) successfully deleted
I20260620 19:13:58.233793 28420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:58.233537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.65:46619 (local address 127.0.0.1:36886)
0620 19:13:58.233659 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:58.233719 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.65:46619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 19:13:58.233958 27935 catalog_manager.cc:4584] TS 577f4e9d83434b5f8fab117d80947d3a (127.9.155.65:46619): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4bb6760c106c4253a4f90112d6a68528: Network error: Client connection negotiation failed: client connection to 127.9.155.65:46619: connect: Connection refused (error 111)
I20260620 19:13:58.237929 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 28279
I20260620 19:13:58.238715 28420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:58.238219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.66:34831 (local address 127.0.0.1:33646)
0620 19:13:58.238573 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:58.238628 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.66:34831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":274}
I20260620 19:13:58.242947 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 27917
2026-06-20T19:13:58Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2190 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-20T19:13:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:13:58Z Disabled control of system clock
I20260620 19:13:58.270885 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982595231230-9837-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982595231230-9837-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982595231230-9837-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:41305
--webserver_interface=127.9.155.126
--webserver_port=0
--builtin_ntp_servers=127.9.155.84:41189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.155.126:41305
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 19:13:58.384102 28454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:13:58.384405 28454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:13:58.384442 28454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:13:58.388922 28454 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:13:58.389007 28454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:13:58.389022 28454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:13:58.389039 28454 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:13:58.389060 28454 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:13:58.394095 28454 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:41189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982595231230-9837-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982595231230-9837-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.155.126:41305
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:41305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.155.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28454
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982595231230-9837-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:13:58.395231 28454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:13:58.396705 28454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:13:58.405606 28460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:13:58.405597 28459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:13:58.405597 28462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:13:58.405620 28454 server_base.cc:1034] running on GCE node
I20260620 19:13:58.406466 28454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:13:58.407598 28454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:13:58.408784 28454 hybrid_clock.cc:648] HybridClock initialized: now 1781982838408768 us; error 35 us; skew 500 ppm
I20260620 19:13:58.410713 28454 webserver.cc:460] Webserver started at http://127.9.155.126:38749/ using document root <none> and password file <none>
I20260620 19:13:58.411273 28454 fs_manager.cc:359] Metadata directory not provided
I20260620 19:13:58.411339 28454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:13:58.411557 28454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:13:58.413655 28454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982595231230-9837-0/minicluster-data/master-0/data/instance:
uuid: "da7b2488f6164ff89cbdd42b3627f025"
format_stamp: "Formatted at 2026-06-20 19:13:58 on dist-test-slave-n3fx"
server_key: "f6b3e804d960171403102d9b7b266452"
server_key_iv: "334f8cad9bd0be53f9096098eb9af5c3"
server_key_version: "encryptionkey@0"
I20260620 19:13:58.414219 28454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982595231230-9837-0/minicluster-data/master-0/wal/instance:
uuid: "da7b2488f6164ff89cbdd42b3627f025"
format_stamp: "Formatted at 2026-06-20 19:13:58 on dist-test-slave-n3fx"
server_key: "f6b3e804d960171403102d9b7b266452"
server_key_iv: "334f8cad9bd0be53f9096098eb9af5c3"
server_key_version: "encryptionkey@0"
I20260620 19:13:58.418108 28454 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260620 19:13:58.420620 28468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:58.421897 28454 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:13:58.422016 28454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982595231230-9837-0/minicluster-data/master-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982595231230-9837-0/minicluster-data/master-0/wal
uuid: "da7b2488f6164ff89cbdd42b3627f025"
format_stamp: "Formatted at 2026-06-20 19:13:58 on dist-test-slave-n3fx"
server_key: "f6b3e804d960171403102d9b7b266452"
server_key_iv: "334f8cad9bd0be53f9096098eb9af5c3"
server_key_version: "encryptionkey@0"
I20260620 19:13:58.422122 28454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982595231230-9837-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982595231230-9837-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982595231230-9837-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:13:58.448994 28454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:13:58.449728 28454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:13:58.449899 28454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:13:58.456282 28454 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.126:41305
I20260620 19:13:58.456311 28520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.126:41305 every 8 connection(s)
I20260620 19:13:58.457224 28454 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
I20260620 19:13:58.460268 28521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:13:58.465749 28521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da7b2488f6164ff89cbdd42b3627f025: Bootstrap starting.
I20260620 19:13:58.467025 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 28454
I20260620 19:13:58.467135 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982595231230-9837-0/minicluster-data/master-0/wal/instance
I20260620 19:13:58.467391 9837 external_mini_cluster.cc:1428] Setting key dc99c22ef34a3d3e293a07b1510c4e78
I20260620 19:13:58.468385 28521 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da7b2488f6164ff89cbdd42b3627f025: Neither blocks nor log segments found. Creating new log.
I20260620 19:13:58.469101 28521 log.cc:824] T 00000000000000000000000000000000 P da7b2488f6164ff89cbdd42b3627f025: Log is configured to *not* fsync() on all Append() calls
I20260620 19:13:58.471020 28521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da7b2488f6164ff89cbdd42b3627f025: No bootstrap required, opened a new log
I20260620 19:13:58.474905 28521 raft_consensus.cc:348] T 00000000000000000000000000000000 P da7b2488f6164ff89cbdd42b3627f025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da7b2488f6164ff89cbdd42b3627f025" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 41305 } }
I20260620 19:13:58.475176 28521 raft_consensus.cc:374] T 00000000000000000000000000000000 P da7b2488f6164ff89cbdd42b3627f025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:13:58.475235 28521 raft_consensus.cc:729] T 00000000000000000000000000000000 P da7b2488f6164ff89cbdd42b3627f025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da7b2488f6164ff89cbdd42b3627f025, State: Initialized, Role: FOLLOWER
I20260620 19:13:58.475677 28521 consensus_queue.cc:260] T 00000000000000000000000000000000 P da7b2488f6164ff89cbdd42b3627f025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da7b2488f6164ff89cbdd42b3627f025" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 41305 } }
I20260620 19:13:58.475805 28521 raft_consensus.cc:388] T 00000000000000000000000000000000 P da7b2488f6164ff89cbdd42b3627f025 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:13:58.475864 28521 raft_consensus.cc:482] T 00000000000000000000000000000000 P da7b2488f6164ff89cbdd42b3627f025 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:13:58.475957 28521 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da7b2488f6164ff89cbdd42b3627f025 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:13:58.476280 28524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:58.468507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41632 (local address 127.9.155.126:41305)
0620 19:13:58.469145 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:58.469158 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:58.469192 (+ 34us) server_negotiation.cc:407] Connection header received
0620 19:13:58.469837 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:58.469853 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:58.470195 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:58.470473 (+ 278us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:58.470811 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:58.471782 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:58.472979 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:58.473486 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:58.473611 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:58.474459 (+ 848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:13:58.474464 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:13:58.474497 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:13:58.474868 (+ 371us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:13:58.474921 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:13:58.474940 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:13:58.475013 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:58.475160 (+ 147us) server_negotiation.cc:299] Negotiation successful
0620 19:13:58.475422 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":131,"threads_started":1}
I20260620 19:13:58.476874 28521 raft_consensus.cc:504] T 00000000000000000000000000000000 P da7b2488f6164ff89cbdd42b3627f025 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da7b2488f6164ff89cbdd42b3627f025" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 41305 } }
I20260620 19:13:58.477191 28521 leader_election.cc:302] T 00000000000000000000000000000000 P da7b2488f6164ff89cbdd42b3627f025 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da7b2488f6164ff89cbdd42b3627f025; no voters:
I20260620 19:13:58.477468 28521 leader_election.cc:288] T 00000000000000000000000000000000 P da7b2488f6164ff89cbdd42b3627f025 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:13:58.477571 28526 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da7b2488f6164ff89cbdd42b3627f025 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:13:58.477800 28526 raft_consensus.cc:686] T 00000000000000000000000000000000 P da7b2488f6164ff89cbdd42b3627f025 [term 1 LEADER]: Becoming Leader. State: Replica: da7b2488f6164ff89cbdd42b3627f025, State: Running, Role: LEADER
I20260620 19:13:58.478119 28526 consensus_queue.cc:237] T 00000000000000000000000000000000 P da7b2488f6164ff89cbdd42b3627f025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da7b2488f6164ff89cbdd42b3627f025" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 41305 } }
I20260620 19:13:58.478672 28521 sys_catalog.cc:564] T 00000000000000000000000000000000 P da7b2488f6164ff89cbdd42b3627f025 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:13:58.479221 28528 sys_catalog.cc:455] T 00000000000000000000000000000000 P da7b2488f6164ff89cbdd42b3627f025 [sys.catalog]: SysCatalogTable state changed. Reason: New leader da7b2488f6164ff89cbdd42b3627f025. Latest consensus state: current_term: 1 leader_uuid: "da7b2488f6164ff89cbdd42b3627f025" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da7b2488f6164ff89cbdd42b3627f025" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 41305 } } }
I20260620 19:13:58.479938 28528 sys_catalog.cc:458] T 00000000000000000000000000000000 P da7b2488f6164ff89cbdd42b3627f025 [sys.catalog]: This master's current role is: LEADER
I20260620 19:13:58.479831 28527 sys_catalog.cc:455] T 00000000000000000000000000000000 P da7b2488f6164ff89cbdd42b3627f025 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da7b2488f6164ff89cbdd42b3627f025" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da7b2488f6164ff89cbdd42b3627f025" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 41305 } } }
I20260620 19:13:58.480180 28527 sys_catalog.cc:458] T 00000000000000000000000000000000 P da7b2488f6164ff89cbdd42b3627f025 [sys.catalog]: This master's current role is: LEADER
I20260620 19:13:58.480830 28536 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:13:58.483731 28536 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:13:58.489392 28536 catalog_manager.cc:1347] Generated new cluster ID: 7c1816b7b1ef468d9f389b0ef60bda2c
I20260620 19:13:58.489470 28536 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:13:58.498786 28536 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:13:58.499704 28536 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:13:58.512924 28536 catalog_manager.cc:5868] T 00000000000000000000000000000000 P da7b2488f6164ff89cbdd42b3627f025: Generated new TSK 0
I20260620 19:13:58.513655 28536 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:13:58.517827 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982595231230-9837-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982595231230-9837-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982595231230-9837-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.65:0
--local_ip_for_outbound_sockets=127.9.155.65
--webserver_interface=127.9.155.65
--webserver_port=0
--tserver_master_addrs=127.9.155.126:41305
--builtin_ntp_servers=127.9.155.84:41189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 19:13:58.642268 28545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:13:58.642540 28545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:13:58.642585 28545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:13:58.646615 28545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:13:58.646674 28545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:13:58.646729 28545 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.65
I20260620 19:13:58.651459 28545 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:41189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982595231230-9837-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982595231230-9837-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.155.65
--webserver_port=0
--tserver_master_addrs=127.9.155.126:41305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28545
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.65
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982595231230-9837-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:13:58.652537 28545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:13:58.653942 28545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:13:58.663533 28551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:13:58.663619 28550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:13:58.663533 28553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:13:58.664047 28545 server_base.cc:1034] running on GCE node
I20260620 19:13:58.664499 28545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:13:58.665156 28545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:13:58.666316 28545 hybrid_clock.cc:648] HybridClock initialized: now 1781982838666297 us; error 33 us; skew 500 ppm
I20260620 19:13:58.668099 28545 webserver.cc:460] Webserver started at http://127.9.155.65:33489/ using document root <none> and password file <none>
I20260620 19:13:58.668696 28545 fs_manager.cc:359] Metadata directory not provided
I20260620 19:13:58.668763 28545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:13:58.668949 28545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:13:58.670950 28545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982595231230-9837-0/minicluster-data/ts-0/data/instance:
uuid: "555af0dc008a4472af78305269d8c363"
format_stamp: "Formatted at 2026-06-20 19:13:58 on dist-test-slave-n3fx"
server_key: "088768074ca1bd0d5e1e1b76be7bc91f"
server_key_iv: "ee599a9d6ab440c724b5526917395490"
server_key_version: "encryptionkey@0"
I20260620 19:13:58.671478 28545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982595231230-9837-0/minicluster-data/ts-0/wal/instance:
uuid: "555af0dc008a4472af78305269d8c363"
format_stamp: "Formatted at 2026-06-20 19:13:58 on dist-test-slave-n3fx"
server_key: "088768074ca1bd0d5e1e1b76be7bc91f"
server_key_iv: "ee599a9d6ab440c724b5526917395490"
server_key_version: "encryptionkey@0"
I20260620 19:13:58.675153 28545 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 19:13:58.677670 28559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:58.679138 28545 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 19:13:58.679251 28545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982595231230-9837-0/minicluster-data/ts-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982595231230-9837-0/minicluster-data/ts-0/wal
uuid: "555af0dc008a4472af78305269d8c363"
format_stamp: "Formatted at 2026-06-20 19:13:58 on dist-test-slave-n3fx"
server_key: "088768074ca1bd0d5e1e1b76be7bc91f"
server_key_iv: "ee599a9d6ab440c724b5526917395490"
server_key_version: "encryptionkey@0"
I20260620 19:13:58.679356 28545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982595231230-9837-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982595231230-9837-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982595231230-9837-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:13:58.702864 28545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:13:58.703562 28545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:13:58.703727 28545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:13:58.704345 28545 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:13:58.705304 28545 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:13:58.705350 28545 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:58.705389 28545 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:13:58.705408 28545 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:58.713738 28545 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.65:34581
I20260620 19:13:58.713836 28672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.65:34581 every 8 connection(s)
I20260620 19:13:58.714664 28545 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
I20260620 19:13:58.723441 28524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:58.716482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.65:54765 (local address 127.9.155.126:41305)
0620 19:13:58.716638 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:58.716642 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:58.717731 (+ 1089us) server_negotiation.cc:407] Connection header received
0620 19:13:58.719026 (+ 1295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:58.719029 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:58.719076 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:58.719155 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:58.719997 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:58.720603 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:58.721729 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:58.721962 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:58.722044 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:58.723045 (+ 1001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:13:58.723049 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:13:58.723051 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:13:58.723064 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:13:58.723084 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:13:58.723092 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:13:58.723158 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:58.723269 (+ 111us) server_negotiation.cc:299] Negotiation successful
0620 19:13:58.723323 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 19:13:58.724352 28675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:58.716908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:41305 (local address 127.9.155.65:54765)
0620 19:13:58.717572 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:58.717613 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:58.718822 (+ 1209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:58.719294 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:58.719300 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:58.719384 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:58.719872 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:58.719884 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:58.720716 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:58.720719 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:58.721614 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:58.721625 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:58.722093 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:58.722096 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:58.722240 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:58.722791 (+ 551us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:13:58.722793 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:58.722831 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:13:58.722892 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:13:58.723164 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:13:58.723169 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:13:58.723173 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:13:58.723263 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 19:13:58.723479 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":107,"threads_started":1}
I20260620 19:13:58.724946 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 28545
I20260620 19:13:58.725078 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982595231230-9837-0/minicluster-data/ts-0/wal/instance
I20260620 19:13:58.725449 9837 external_mini_cluster.cc:1428] Setting key 22ad422d668b97277434315c9451e335
I20260620 19:13:58.725692 28673 heartbeater.cc:347] Connected to a master server at 127.9.155.126:41305
I20260620 19:13:58.725927 28673 heartbeater.cc:464] Registering TS with master...
I20260620 19:13:58.726518 28673 heartbeater.cc:511] Master 127.9.155.126:41305 requested a full tablet report, sending...
I20260620 19:13:58.727314 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982595231230-9837-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982595231230-9837-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982595231230-9837-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.66:0
--local_ip_for_outbound_sockets=127.9.155.66
--webserver_interface=127.9.155.66
--webserver_port=0
--tserver_master_addrs=127.9.155.126:41305
--builtin_ntp_servers=127.9.155.84:41189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 19:13:58.728060 28485 ts_manager.cc:194] Registered new tserver with Master: 555af0dc008a4472af78305269d8c363 (127.9.155.65:34581)
I20260620 19:13:58.729214 28485 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.155.65:54765
W20260620 19:13:58.834069 28676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:13:58.834350 28676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:13:58.834389 28676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:13:58.838769 28676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:13:58.838852 28676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:13:58.838912 28676 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.66
I20260620 19:13:58.843852 28676 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:41189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982595231230-9837-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982595231230-9837-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.155.66
--webserver_port=0
--tserver_master_addrs=127.9.155.126:41305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28676
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.66
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982595231230-9837-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:13:58.844935 28676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:13:58.846251 28676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:13:58.855821 28682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:13:58.855978 28676 server_base.cc:1034] running on GCE node
W20260620 19:13:58.855821 28681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:13:58.855821 28684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:13:58.856523 28676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:13:58.857172 28676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:13:58.858350 28676 hybrid_clock.cc:648] HybridClock initialized: now 1781982838858326 us; error 37 us; skew 500 ppm
I20260620 19:13:58.860530 28676 webserver.cc:460] Webserver started at http://127.9.155.66:35219/ using document root <none> and password file <none>
I20260620 19:13:58.861083 28676 fs_manager.cc:359] Metadata directory not provided
I20260620 19:13:58.861155 28676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:13:58.861342 28676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:13:58.863359 28676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982595231230-9837-0/minicluster-data/ts-1/data/instance:
uuid: "4a12eae723184d198ee0780a913b3aef"
format_stamp: "Formatted at 2026-06-20 19:13:58 on dist-test-slave-n3fx"
server_key: "eec96054312a55edbadaefc09489e6d1"
server_key_iv: "19bb72aa2fb2d47bb25d0a237bc7e576"
server_key_version: "encryptionkey@0"
I20260620 19:13:58.863890 28676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982595231230-9837-0/minicluster-data/ts-1/wal/instance:
uuid: "4a12eae723184d198ee0780a913b3aef"
format_stamp: "Formatted at 2026-06-20 19:13:58 on dist-test-slave-n3fx"
server_key: "eec96054312a55edbadaefc09489e6d1"
server_key_iv: "19bb72aa2fb2d47bb25d0a237bc7e576"
server_key_version: "encryptionkey@0"
I20260620 19:13:58.867707 28676 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 19:13:58.870229 28690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:58.871565 28676 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 19:13:58.871690 28676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982595231230-9837-0/minicluster-data/ts-1/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982595231230-9837-0/minicluster-data/ts-1/wal
uuid: "4a12eae723184d198ee0780a913b3aef"
format_stamp: "Formatted at 2026-06-20 19:13:58 on dist-test-slave-n3fx"
server_key: "eec96054312a55edbadaefc09489e6d1"
server_key_iv: "19bb72aa2fb2d47bb25d0a237bc7e576"
server_key_version: "encryptionkey@0"
I20260620 19:13:58.871810 28676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982595231230-9837-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982595231230-9837-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982595231230-9837-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:13:58.893451 28676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:13:58.894191 28676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:13:58.894372 28676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:13:58.895020 28676 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:13:58.895985 28676 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:13:58.896034 28676 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:58.896086 28676 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:13:58.896122 28676 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:58.904755 28676 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.66:40603
I20260620 19:13:58.904781 28803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.66:40603 every 8 connection(s)
I20260620 19:13:58.905637 28676 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
I20260620 19:13:58.913735 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 28676
I20260620 19:13:58.913847 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982595231230-9837-0/minicluster-data/ts-1/wal/instance
I20260620 19:13:58.914136 9837 external_mini_cluster.cc:1428] Setting key c4e34a7e1b007fc790f0c5eabea3ccfb
I20260620 19:13:58.915040 28524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:58.907610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.66:33181 (local address 127.9.155.126:41305)
0620 19:13:58.907759 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:58.907762 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:58.908810 (+ 1048us) server_negotiation.cc:407] Connection header received
0620 19:13:58.910224 (+ 1414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:58.910227 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:58.910278 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:58.910356 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:58.911290 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:58.911829 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:58.913030 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:58.913259 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:58.913342 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:58.914514 (+ 1172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:13:58.914518 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:13:58.914520 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:13:58.914534 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:13:58.914553 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:13:58.914561 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:13:58.914635 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:58.914821 (+ 186us) server_negotiation.cc:299] Negotiation successful
0620 19:13:58.914890 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 19:13:58.916064 28806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:58.907979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:41305 (local address 127.9.155.66:33181)
0620 19:13:58.908662 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:58.908704 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:58.910016 (+ 1312us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:58.910566 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:58.910574 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:58.910660 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:58.911171 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:58.911185 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:58.911942 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:58.911945 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:58.912901 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:58.912911 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:58.913403 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:58.913406 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:58.913580 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:58.914227 (+ 647us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:13:58.914229 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:58.914286 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:13:58.914358 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:13:58.914650 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:13:58.914660 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:13:58.914666 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 19:13:58.914785 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 19:13:58.915128 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":102,"threads_started":1}
I20260620 19:13:58.916378 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982595231230-9837-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982595231230-9837-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982595231230-9837-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.67:0
--local_ip_for_outbound_sockets=127.9.155.67
--webserver_interface=127.9.155.67
--webserver_port=0
--tserver_master_addrs=127.9.155.126:41305
--builtin_ntp_servers=127.9.155.84:41189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 19:13:58.917184 28804 heartbeater.cc:347] Connected to a master server at 127.9.155.126:41305
I20260620 19:13:58.917514 28804 heartbeater.cc:464] Registering TS with master...
I20260620 19:13:58.918169 28804 heartbeater.cc:511] Master 127.9.155.126:41305 requested a full tablet report, sending...
I20260620 19:13:58.919284 28485 ts_manager.cc:194] Registered new tserver with Master: 4a12eae723184d198ee0780a913b3aef (127.9.155.66:40603)
I20260620 19:13:58.919847 28485 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.155.66:33181
W20260620 19:13:59.026069 28807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:13:59.026329 28807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:13:59.026382 28807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:13:59.030422 28807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:13:59.030488 28807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:13:59.030570 28807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.67
I20260620 19:13:59.035437 28807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:41189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982595231230-9837-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982595231230-9837-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.155.67
--webserver_port=0
--tserver_master_addrs=127.9.155.126:41305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.67
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982595231230-9837-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:13:59.036576 28807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:13:59.037909 28807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:13:59.047441 28812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:13:59.047457 28813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:13:59.047435 28815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:13:59.047668 28807 server_base.cc:1034] running on GCE node
I20260620 19:13:59.048120 28807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:13:59.048791 28807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:13:59.049973 28807 hybrid_clock.cc:648] HybridClock initialized: now 1781982839049954 us; error 29 us; skew 500 ppm
I20260620 19:13:59.051839 28807 webserver.cc:460] Webserver started at http://127.9.155.67:42265/ using document root <none> and password file <none>
I20260620 19:13:59.052454 28807 fs_manager.cc:359] Metadata directory not provided
I20260620 19:13:59.052525 28807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:13:59.052695 28807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:13:59.054682 28807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982595231230-9837-0/minicluster-data/ts-2/data/instance:
uuid: "d22eaf889db749c8b6f5469f69fa6379"
format_stamp: "Formatted at 2026-06-20 19:13:59 on dist-test-slave-n3fx"
server_key: "12a0117bd440bc4cbc8d833b4fdefc6f"
server_key_iv: "d2fab97f9a2bbcd68c804940368ec9d5"
server_key_version: "encryptionkey@0"
I20260620 19:13:59.055179 28807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982595231230-9837-0/minicluster-data/ts-2/wal/instance:
uuid: "d22eaf889db749c8b6f5469f69fa6379"
format_stamp: "Formatted at 2026-06-20 19:13:59 on dist-test-slave-n3fx"
server_key: "12a0117bd440bc4cbc8d833b4fdefc6f"
server_key_iv: "d2fab97f9a2bbcd68c804940368ec9d5"
server_key_version: "encryptionkey@0"
I20260620 19:13:59.058985 28807 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 19:13:59.061535 28821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:59.062776 28807 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 19:13:59.062876 28807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982595231230-9837-0/minicluster-data/ts-2/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982595231230-9837-0/minicluster-data/ts-2/wal
uuid: "d22eaf889db749c8b6f5469f69fa6379"
format_stamp: "Formatted at 2026-06-20 19:13:59 on dist-test-slave-n3fx"
server_key: "12a0117bd440bc4cbc8d833b4fdefc6f"
server_key_iv: "d2fab97f9a2bbcd68c804940368ec9d5"
server_key_version: "encryptionkey@0"
I20260620 19:13:59.062984 28807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982595231230-9837-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982595231230-9837-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982595231230-9837-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:13:59.107301 28807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:13:59.107998 28807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:13:59.108165 28807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:13:59.108773 28807 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:13:59.109679 28807 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:13:59.109723 28807 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:59.109763 28807 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:13:59.109791 28807 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:59.118175 28807 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.67:34545
I20260620 19:13:59.118211 28934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.67:34545 every 8 connection(s)
I20260620 19:13:59.119086 28807 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
I20260620 19:13:59.123062 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 28807
I20260620 19:13:59.123170 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982595231230-9837-0/minicluster-data/ts-2/wal/instance
I20260620 19:13:59.123463 9837 external_mini_cluster.cc:1428] Setting key 388a3b51fe6a966696a7a91165f4d645
I20260620 19:13:59.128911 28524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:59.121034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.67:37365 (local address 127.9.155.126:41305)
0620 19:13:59.121180 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:59.121185 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:59.122188 (+ 1003us) server_negotiation.cc:407] Connection header received
0620 19:13:59.123777 (+ 1589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:59.123782 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:59.123852 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:59.123949 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:59.125102 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:59.125830 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:59.127094 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:59.127318 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:59.127406 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:59.128538 (+ 1132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:13:59.128542 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:13:59.128544 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:13:59.128556 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:13:59.128575 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:13:59.128582 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:13:59.128642 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:59.128753 (+ 111us) server_negotiation.cc:299] Negotiation successful
0620 19:13:59.128803 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 19:13:59.129751 28937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:59.121397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:41305 (local address 127.9.155.67:37365)
0620 19:13:59.122037 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:59.122078 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:59.123535 (+ 1457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:59.124170 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:59.124180 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:59.124313 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:59.124970 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:59.124986 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:59.125955 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:59.125958 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:59.126945 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:59.126955 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:59.127448 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:59.127451 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:59.127594 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:59.128299 (+ 705us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:13:59.128301 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:59.128341 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:13:59.128402 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:13:59.128650 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:13:59.128655 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:13:59.128659 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:13:59.128749 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 19:13:59.128970 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":111,"threads_started":1}
I20260620 19:13:59.130796 28935 heartbeater.cc:347] Connected to a master server at 127.9.155.126:41305
I20260620 19:13:59.131011 28935 heartbeater.cc:464] Registering TS with master...
I20260620 19:13:59.131485 28935 heartbeater.cc:511] Master 127.9.155.126:41305 requested a full tablet report, sending...
I20260620 19:13:59.132507 28485 ts_manager.cc:194] Registered new tserver with Master: d22eaf889db749c8b6f5469f69fa6379 (127.9.155.67:34545)
I20260620 19:13:59.133054 28485 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.155.67:37365
I20260620 19:13:59.137344 9837 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:13:59.143146 28524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:59.139204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41648 (local address 127.9.155.126:41305)
0620 19:13:59.139408 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:59.139412 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:59.139530 (+ 118us) server_negotiation.cc:407] Connection header received
0620 19:13:59.139652 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:59.139655 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:59.139695 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:59.139754 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:59.140079 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:59.140631 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:59.141540 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:59.141752 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:59.141837 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:59.142661 (+ 824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:13:59.142664 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:13:59.142666 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:13:59.142679 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:13:59.142697 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:13:59.142705 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:13:59.142807 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:59.142953 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 19:13:59.142994 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260620 19:13:59.146170 28485 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41648:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:13:59.149195 28485 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:13:59.163655 28949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:59.158053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.66:40603 (local address 127.0.0.1:51732)
0620 19:13:59.158267 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:59.158280 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:59.158362 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:59.159791 (+ 1429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:59.159793 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:59.159809 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:13:59.159883 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:59.159889 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:59.160863 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:59.160866 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:59.162299 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:59.162310 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:59.163256 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:59.163260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:59.163340 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:59.163361 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 19:13:59.163418 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 19:13:59.163489 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":189,"thread_start_us":88,"threads_started":1}
I20260620 19:13:59.163717 28946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:59.157335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.67:34545 (local address 127.0.0.1:33154)
0620 19:13:59.157710 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:59.157756 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:59.157882 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:59.159152 (+ 1270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:59.159156 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:59.159191 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:13:59.159313 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:59.159318 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:59.160850 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:59.160853 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:59.162327 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:59.162335 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:59.163357 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:59.163360 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:59.163440 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:59.163460 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 19:13:59.163526 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 19:13:59.163591 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":119,"spinlock_wait_cycles":7168,"thread_start_us":82,"threads_started":1}
I20260620 19:13:59.163736 28951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:59.158223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51732 (local address 127.9.155.66:40603)
0620 19:13:59.159447 (+ 1224us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:59.159457 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:59.159475 (+ 18us) server_negotiation.cc:407] Connection header received
0620 19:13:59.159532 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:59.159535 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:59.159672 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:59.159791 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:13:59.159983 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:59.160752 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:59.162436 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:59.163158 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:59.163256 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:59.163435 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:59.163484 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 19:13:59.163594 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1121,"thread_start_us":134,"threads_started":1}
I20260620 19:13:59.164324 28950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:59.158131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51378 (local address 127.9.155.65:34581)
0620 19:13:59.158719 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:59.158731 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:59.158750 (+ 19us) server_negotiation.cc:407] Connection header received
0620 19:13:59.158809 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:59.158813 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:59.158992 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:59.159136 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:13:59.159438 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:59.160693 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:59.162010 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:59.162673 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:59.162771 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:59.162965 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:59.163571 (+ 606us) server_negotiation.cc:299] Negotiation successful
0620 19:13:59.163681 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":105,"threads_started":1}
I20260620 19:13:59.164408 28947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:59.157536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33154 (local address 127.9.155.67:34545)
0620 19:13:59.158626 (+ 1090us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:59.158637 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:59.158663 (+ 26us) server_negotiation.cc:407] Connection header received
0620 19:13:59.158742 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:59.158748 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:59.158990 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:59.159449 (+ 459us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:13:59.159603 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:59.160725 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:59.162494 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:59.163238 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:59.163912 (+ 674us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:59.164089 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:59.164146 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 19:13:59.164269 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":974,"thread_start_us":169,"threads_started":1}
I20260620 19:13:59.164551 28948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:59.157721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.65:34581 (local address 127.0.0.1:51378)
0620 19:13:59.157976 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:59.157989 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:59.158077 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:59.159144 (+ 1067us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:59.159150 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:59.159182 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:13:59.159314 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:59.159320 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:59.160821 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:59.160825 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:59.161889 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:59.161900 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:59.163026 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:59.163029 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:59.163146 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:59.163188 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 19:13:59.163298 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 19:13:59.163428 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":53,"spinlock_wait_cycles":3968,"thread_start_us":103,"threads_started":1}
I20260620 19:13:59.167011 28607 tablet_service.cc:1467] Processing CreateTablet for tablet a84b1295cde84ecebfd7625ac7d91ac5 (DEFAULT_TABLE table=test-table [id=b42242ad460f472aafd4936f2d115b1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:13:59.166972 28869 tablet_service.cc:1467] Processing CreateTablet for tablet a84b1295cde84ecebfd7625ac7d91ac5 (DEFAULT_TABLE table=test-table [id=b42242ad460f472aafd4936f2d115b1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:13:59.166905 28738 tablet_service.cc:1467] Processing CreateTablet for tablet a84b1295cde84ecebfd7625ac7d91ac5 (DEFAULT_TABLE table=test-table [id=b42242ad460f472aafd4936f2d115b1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:13:59.167940 28607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a84b1295cde84ecebfd7625ac7d91ac5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:13:59.167963 28738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a84b1295cde84ecebfd7625ac7d91ac5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:13:59.167961 28869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a84b1295cde84ecebfd7625ac7d91ac5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:13:59.174273 28952 tablet_bootstrap.cc:492] T a84b1295cde84ecebfd7625ac7d91ac5 P 555af0dc008a4472af78305269d8c363: Bootstrap starting.
I20260620 19:13:59.175457 28953 tablet_bootstrap.cc:492] T a84b1295cde84ecebfd7625ac7d91ac5 P d22eaf889db749c8b6f5469f69fa6379: Bootstrap starting.
I20260620 19:13:59.176039 28954 tablet_bootstrap.cc:492] T a84b1295cde84ecebfd7625ac7d91ac5 P 4a12eae723184d198ee0780a913b3aef: Bootstrap starting.
I20260620 19:13:59.177050 28952 tablet_bootstrap.cc:654] T a84b1295cde84ecebfd7625ac7d91ac5 P 555af0dc008a4472af78305269d8c363: Neither blocks nor log segments found. Creating new log.
I20260620 19:13:59.177392 28953 tablet_bootstrap.cc:654] T a84b1295cde84ecebfd7625ac7d91ac5 P d22eaf889db749c8b6f5469f69fa6379: Neither blocks nor log segments found. Creating new log.
I20260620 19:13:59.177922 28952 log.cc:824] T a84b1295cde84ecebfd7625ac7d91ac5 P 555af0dc008a4472af78305269d8c363: Log is configured to *not* fsync() on all Append() calls
I20260620 19:13:59.178035 28953 log.cc:824] T a84b1295cde84ecebfd7625ac7d91ac5 P d22eaf889db749c8b6f5469f69fa6379: Log is configured to *not* fsync() on all Append() calls
I20260620 19:13:59.178088 28954 tablet_bootstrap.cc:654] T a84b1295cde84ecebfd7625ac7d91ac5 P 4a12eae723184d198ee0780a913b3aef: Neither blocks nor log segments found. Creating new log.
I20260620 19:13:59.178834 28954 log.cc:824] T a84b1295cde84ecebfd7625ac7d91ac5 P 4a12eae723184d198ee0780a913b3aef: Log is configured to *not* fsync() on all Append() calls
I20260620 19:13:59.179994 28953 tablet_bootstrap.cc:492] T a84b1295cde84ecebfd7625ac7d91ac5 P d22eaf889db749c8b6f5469f69fa6379: No bootstrap required, opened a new log
I20260620 19:13:59.179988 28952 tablet_bootstrap.cc:492] T a84b1295cde84ecebfd7625ac7d91ac5 P 555af0dc008a4472af78305269d8c363: No bootstrap required, opened a new log
I20260620 19:13:59.180184 28952 ts_tablet_manager.cc:1397] T a84b1295cde84ecebfd7625ac7d91ac5 P 555af0dc008a4472af78305269d8c363: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 19:13:59.180184 28953 ts_tablet_manager.cc:1397] T a84b1295cde84ecebfd7625ac7d91ac5 P d22eaf889db749c8b6f5469f69fa6379: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 19:13:59.180541 28954 tablet_bootstrap.cc:492] T a84b1295cde84ecebfd7625ac7d91ac5 P 4a12eae723184d198ee0780a913b3aef: No bootstrap required, opened a new log
I20260620 19:13:59.180719 28954 ts_tablet_manager.cc:1397] T a84b1295cde84ecebfd7625ac7d91ac5 P 4a12eae723184d198ee0780a913b3aef: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260620 19:13:59.184142 28954 raft_consensus.cc:348] T a84b1295cde84ecebfd7625ac7d91ac5 P 4a12eae723184d198ee0780a913b3aef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d22eaf889db749c8b6f5469f69fa6379" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34545 } } peers { permanent_uuid: "4a12eae723184d198ee0780a913b3aef" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40603 } } peers { permanent_uuid: "555af0dc008a4472af78305269d8c363" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 34581 } }
I20260620 19:13:59.184399 28954 raft_consensus.cc:374] T a84b1295cde84ecebfd7625ac7d91ac5 P 4a12eae723184d198ee0780a913b3aef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:13:59.184451 28954 raft_consensus.cc:729] T a84b1295cde84ecebfd7625ac7d91ac5 P 4a12eae723184d198ee0780a913b3aef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a12eae723184d198ee0780a913b3aef, State: Initialized, Role: FOLLOWER
I20260620 19:13:59.184358 28952 raft_consensus.cc:348] T a84b1295cde84ecebfd7625ac7d91ac5 P 555af0dc008a4472af78305269d8c363 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d22eaf889db749c8b6f5469f69fa6379" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34545 } } peers { permanent_uuid: "4a12eae723184d198ee0780a913b3aef" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40603 } } peers { permanent_uuid: "555af0dc008a4472af78305269d8c363" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 34581 } }
I20260620 19:13:59.184614 28952 raft_consensus.cc:374] T a84b1295cde84ecebfd7625ac7d91ac5 P 555af0dc008a4472af78305269d8c363 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:13:59.184677 28952 raft_consensus.cc:729] T a84b1295cde84ecebfd7625ac7d91ac5 P 555af0dc008a4472af78305269d8c363 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 555af0dc008a4472af78305269d8c363, State: Initialized, Role: FOLLOWER
I20260620 19:13:59.184804 28953 raft_consensus.cc:348] T a84b1295cde84ecebfd7625ac7d91ac5 P d22eaf889db749c8b6f5469f69fa6379 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d22eaf889db749c8b6f5469f69fa6379" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34545 } } peers { permanent_uuid: "4a12eae723184d198ee0780a913b3aef" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40603 } } peers { permanent_uuid: "555af0dc008a4472af78305269d8c363" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 34581 } }
I20260620 19:13:59.184914 28954 consensus_queue.cc:260] T a84b1295cde84ecebfd7625ac7d91ac5 P 4a12eae723184d198ee0780a913b3aef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d22eaf889db749c8b6f5469f69fa6379" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34545 } } peers { permanent_uuid: "4a12eae723184d198ee0780a913b3aef" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40603 } } peers { permanent_uuid: "555af0dc008a4472af78305269d8c363" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 34581 } }
I20260620 19:13:59.185027 28953 raft_consensus.cc:374] T a84b1295cde84ecebfd7625ac7d91ac5 P d22eaf889db749c8b6f5469f69fa6379 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:13:59.185096 28953 raft_consensus.cc:729] T a84b1295cde84ecebfd7625ac7d91ac5 P d22eaf889db749c8b6f5469f69fa6379 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d22eaf889db749c8b6f5469f69fa6379, State: Initialized, Role: FOLLOWER
I20260620 19:13:59.185070 28952 consensus_queue.cc:260] T a84b1295cde84ecebfd7625ac7d91ac5 P 555af0dc008a4472af78305269d8c363 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d22eaf889db749c8b6f5469f69fa6379" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34545 } } peers { permanent_uuid: "4a12eae723184d198ee0780a913b3aef" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40603 } } peers { permanent_uuid: "555af0dc008a4472af78305269d8c363" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 34581 } }
I20260620 19:13:59.185544 28804 heartbeater.cc:503] Master 127.9.155.126:41305 was elected leader, sending a full tablet report...
I20260620 19:13:59.185487 28953 consensus_queue.cc:260] T a84b1295cde84ecebfd7625ac7d91ac5 P d22eaf889db749c8b6f5469f69fa6379 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d22eaf889db749c8b6f5469f69fa6379" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34545 } } peers { permanent_uuid: "4a12eae723184d198ee0780a913b3aef" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40603 } } peers { permanent_uuid: "555af0dc008a4472af78305269d8c363" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 34581 } }
I20260620 19:13:59.185886 28954 ts_tablet_manager.cc:1428] T a84b1295cde84ecebfd7625ac7d91ac5 P 4a12eae723184d198ee0780a913b3aef: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20260620 19:13:59.185992 28952 ts_tablet_manager.cc:1428] T a84b1295cde84ecebfd7625ac7d91ac5 P 555af0dc008a4472af78305269d8c363: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 19:13:59.186259 28953 ts_tablet_manager.cc:1428] T a84b1295cde84ecebfd7625ac7d91ac5 P d22eaf889db749c8b6f5469f69fa6379: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 19:13:59.186317 28673 heartbeater.cc:503] Master 127.9.155.126:41305 was elected leader, sending a full tablet report...
I20260620 19:13:59.186550 28935 heartbeater.cc:503] Master 127.9.155.126:41305 was elected leader, sending a full tablet report...
W20260620 19:13:59.215906 28674 tablet.cc:2369] T a84b1295cde84ecebfd7625ac7d91ac5 P 555af0dc008a4472af78305269d8c363: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:13:59.227679 28960 raft_consensus.cc:482] T a84b1295cde84ecebfd7625ac7d91ac5 P d22eaf889db749c8b6f5469f69fa6379 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:13:59.227896 28960 raft_consensus.cc:504] T a84b1295cde84ecebfd7625ac7d91ac5 P d22eaf889db749c8b6f5469f69fa6379 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d22eaf889db749c8b6f5469f69fa6379" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34545 } } peers { permanent_uuid: "4a12eae723184d198ee0780a913b3aef" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40603 } } peers { permanent_uuid: "555af0dc008a4472af78305269d8c363" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 34581 } }
I20260620 19:13:59.229182 28960 leader_election.cc:288] T a84b1295cde84ecebfd7625ac7d91ac5 P d22eaf889db749c8b6f5469f69fa6379 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a12eae723184d198ee0780a913b3aef (127.9.155.66:40603), 555af0dc008a4472af78305269d8c363 (127.9.155.65:34581)
I20260620 19:13:59.232945 28951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:59.229506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.67:51199 (local address 127.9.155.66:40603)
0620 19:13:59.229622 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:59.229626 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:59.229711 (+ 85us) server_negotiation.cc:407] Connection header received
0620 19:13:59.229879 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:59.229882 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:59.229928 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:59.229988 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:13:59.230377 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:59.230871 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:59.231921 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:59.232381 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:59.232488 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:59.232543 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:59.232748 (+ 205us) server_negotiation.cc:299] Negotiation successful
0620 19:13:59.232820 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 19:13:59.232964 28937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:59.229412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.66:40603 (local address 127.9.155.67:51199)
0620 19:13:59.229639 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:59.229653 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:59.229756 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:59.230008 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:59.230011 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:59.230026 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:13:59.230182 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:59.230235 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:59.230977 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:59.230980 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:59.231803 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:59.231812 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:59.232488 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:59.232492 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:59.232621 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:59.232660 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 19:13:59.232723 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 19:13:59.232812 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":50}
I20260620 19:13:59.232969 28950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:59.229554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.67:44061 (local address 127.9.155.65:34581)
0620 19:13:59.229681 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:59.229686 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:59.229905 (+ 219us) server_negotiation.cc:407] Connection header received
0620 19:13:59.230113 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:59.230116 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:59.230169 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:59.230239 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:13:59.230495 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:59.230999 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:59.232000 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:59.232415 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:59.232522 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:59.232570 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:59.232770 (+ 200us) server_negotiation.cc:299] Negotiation successful
0620 19:13:59.232840 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 19:13:59.233179 28961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:59.229465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.65:34581 (local address 127.9.155.67:44061)
0620 19:13:59.229815 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:59.229826 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:59.229935 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:59.230249 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:59.230252 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:59.230267 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:13:59.230385 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:59.230390 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:59.231096 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:59.231099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:59.231884 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:59.231892 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:59.232518 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:59.232521 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:59.232662 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:59.232680 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 19:13:59.232745 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 19:13:59.232837 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"mutex_wait_us":69,"thread_start_us":131,"threads_started":1}
I20260620 19:13:59.233661 28758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a84b1295cde84ecebfd7625ac7d91ac5" candidate_uuid: "d22eaf889db749c8b6f5469f69fa6379" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a12eae723184d198ee0780a913b3aef" is_pre_election: true
I20260620 19:13:59.233757 28627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a84b1295cde84ecebfd7625ac7d91ac5" candidate_uuid: "d22eaf889db749c8b6f5469f69fa6379" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "555af0dc008a4472af78305269d8c363" is_pre_election: true
I20260620 19:13:59.233953 28758 raft_consensus.cc:2454] T a84b1295cde84ecebfd7625ac7d91ac5 P 4a12eae723184d198ee0780a913b3aef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d22eaf889db749c8b6f5469f69fa6379 in term 0.
I20260620 19:13:59.234011 28627 raft_consensus.cc:2454] T a84b1295cde84ecebfd7625ac7d91ac5 P 555af0dc008a4472af78305269d8c363 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d22eaf889db749c8b6f5469f69fa6379 in term 0.
I20260620 19:13:59.234473 28823 leader_election.cc:302] T a84b1295cde84ecebfd7625ac7d91ac5 P d22eaf889db749c8b6f5469f69fa6379 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4a12eae723184d198ee0780a913b3aef, 555af0dc008a4472af78305269d8c363, d22eaf889db749c8b6f5469f69fa6379; no voters:
I20260620 19:13:59.234747 28960 raft_consensus.cc:2781] T a84b1295cde84ecebfd7625ac7d91ac5 P d22eaf889db749c8b6f5469f69fa6379 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:13:59.234833 28960 raft_consensus.cc:482] T a84b1295cde84ecebfd7625ac7d91ac5 P d22eaf889db749c8b6f5469f69fa6379 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:13:59.234861 28960 raft_consensus.cc:3037] T a84b1295cde84ecebfd7625ac7d91ac5 P d22eaf889db749c8b6f5469f69fa6379 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:13:59.235864 28960 raft_consensus.cc:504] T a84b1295cde84ecebfd7625ac7d91ac5 P d22eaf889db749c8b6f5469f69fa6379 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d22eaf889db749c8b6f5469f69fa6379" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34545 } } peers { permanent_uuid: "4a12eae723184d198ee0780a913b3aef" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40603 } } peers { permanent_uuid: "555af0dc008a4472af78305269d8c363" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 34581 } }
I20260620 19:13:59.236207 28960 leader_election.cc:288] T a84b1295cde84ecebfd7625ac7d91ac5 P d22eaf889db749c8b6f5469f69fa6379 [CANDIDATE]: Term 1 election: Requested vote from peers 4a12eae723184d198ee0780a913b3aef (127.9.155.66:40603), 555af0dc008a4472af78305269d8c363 (127.9.155.65:34581)
I20260620 19:13:59.236593 28758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a84b1295cde84ecebfd7625ac7d91ac5" candidate_uuid: "d22eaf889db749c8b6f5469f69fa6379" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a12eae723184d198ee0780a913b3aef"
I20260620 19:13:59.236663 28627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a84b1295cde84ecebfd7625ac7d91ac5" candidate_uuid: "d22eaf889db749c8b6f5469f69fa6379" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "555af0dc008a4472af78305269d8c363"
I20260620 19:13:59.236728 28758 raft_consensus.cc:3037] T a84b1295cde84ecebfd7625ac7d91ac5 P 4a12eae723184d198ee0780a913b3aef [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:13:59.236801 28627 raft_consensus.cc:3037] T a84b1295cde84ecebfd7625ac7d91ac5 P 555af0dc008a4472af78305269d8c363 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:13:59.237926 28758 raft_consensus.cc:2454] T a84b1295cde84ecebfd7625ac7d91ac5 P 4a12eae723184d198ee0780a913b3aef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d22eaf889db749c8b6f5469f69fa6379 in term 1.
I20260620 19:13:59.237958 28627 raft_consensus.cc:2454] T a84b1295cde84ecebfd7625ac7d91ac5 P 555af0dc008a4472af78305269d8c363 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d22eaf889db749c8b6f5469f69fa6379 in term 1.
I20260620 19:13:59.238247 28824 leader_election.cc:302] T a84b1295cde84ecebfd7625ac7d91ac5 P d22eaf889db749c8b6f5469f69fa6379 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a12eae723184d198ee0780a913b3aef, d22eaf889db749c8b6f5469f69fa6379; no voters:
I20260620 19:13:59.238420 28960 raft_consensus.cc:2781] T a84b1295cde84ecebfd7625ac7d91ac5 P d22eaf889db749c8b6f5469f69fa6379 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:13:59.238629 28960 raft_consensus.cc:686] T a84b1295cde84ecebfd7625ac7d91ac5 P d22eaf889db749c8b6f5469f69fa6379 [term 1 LEADER]: Becoming Leader. State: Replica: d22eaf889db749c8b6f5469f69fa6379, State: Running, Role: LEADER
I20260620 19:13:59.238993 28960 consensus_queue.cc:237] T a84b1295cde84ecebfd7625ac7d91ac5 P d22eaf889db749c8b6f5469f69fa6379 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d22eaf889db749c8b6f5469f69fa6379" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34545 } } peers { permanent_uuid: "4a12eae723184d198ee0780a913b3aef" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40603 } } peers { permanent_uuid: "555af0dc008a4472af78305269d8c363" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 34581 } }
I20260620 19:13:59.242218 28483 catalog_manager.cc:5495] T a84b1295cde84ecebfd7625ac7d91ac5 P d22eaf889db749c8b6f5469f69fa6379 reported cstate change: term changed from 0 to 1, leader changed from <none> to d22eaf889db749c8b6f5469f69fa6379 (127.9.155.67). New cstate: current_term: 1 leader_uuid: "d22eaf889db749c8b6f5469f69fa6379" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d22eaf889db749c8b6f5469f69fa6379" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a12eae723184d198ee0780a913b3aef" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "555af0dc008a4472af78305269d8c363" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 34581 } health_report { overall_health: UNKNOWN } } }
I20260620 19:13:59.264873 28947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:59.261341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33170 (local address 127.9.155.67:34545)
0620 19:13:59.261469 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:59.261473 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:59.261505 (+ 32us) server_negotiation.cc:407] Connection header received
0620 19:13:59.261593 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:59.261596 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:59.261656 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:59.261742 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:13:59.261983 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:59.262496 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:59.263350 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:59.263575 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:59.263664 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:59.263982 (+ 318us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:13:59.264438 (+ 456us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:13:59.264570 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:59.264681 (+ 111us) server_negotiation.cc:299] Negotiation successful
0620 19:13:59.264718 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 19:13:59.270778 28758 raft_consensus.cc:1261] T a84b1295cde84ecebfd7625ac7d91ac5 P 4a12eae723184d198ee0780a913b3aef [term 1 FOLLOWER]: Refusing update from remote peer d22eaf889db749c8b6f5469f69fa6379: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:13:59.270787 28627 raft_consensus.cc:1261] T a84b1295cde84ecebfd7625ac7d91ac5 P 555af0dc008a4472af78305269d8c363 [term 1 FOLLOWER]: Refusing update from remote peer d22eaf889db749c8b6f5469f69fa6379: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:13:59.271456 28960 consensus_queue.cc:1035] T a84b1295cde84ecebfd7625ac7d91ac5 P d22eaf889db749c8b6f5469f69fa6379 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a12eae723184d198ee0780a913b3aef" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 40603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:13:59.271641 28962 consensus_queue.cc:1035] T a84b1295cde84ecebfd7625ac7d91ac5 P d22eaf889db749c8b6f5469f69fa6379 [LEADER]: Connected to new peer: Peer: permanent_uuid: "555af0dc008a4472af78305269d8c363" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 34581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:13:59.281302 28967 mvcc.cc:205] Tried to move back new op lower bound from 7299001709646389248 to 7299001709525667840. Current Snapshot: MvccSnapshot[applied={T|T < 7299001709646389248 or (T in {7299001709646389248})}]
I20260620 19:13:59.289974 28484 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41648:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:13:59.290158 28484 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41648:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:13:59.292553 28484 catalog_manager.cc:5782] T 00000000000000000000000000000000 P da7b2488f6164ff89cbdd42b3627f025: Sending DeleteTablet for 3 replicas of tablet a84b1295cde84ecebfd7625ac7d91ac5
I20260620 19:13:59.293517 28607 tablet_service.cc:1514] Processing DeleteTablet for tablet a84b1295cde84ecebfd7625ac7d91ac5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:13:59 UTC) from {username='slave'} at 127.0.0.1:51378
I20260620 19:13:59.293514 28869 tablet_service.cc:1514] Processing DeleteTablet for tablet a84b1295cde84ecebfd7625ac7d91ac5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:13:59 UTC) from {username='slave'} at 127.0.0.1:33154
I20260620 19:13:59.293509 28738 tablet_service.cc:1514] Processing DeleteTablet for tablet a84b1295cde84ecebfd7625ac7d91ac5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:13:59 UTC) from {username='slave'} at 127.0.0.1:51732
I20260620 19:13:59.294142 28977 tablet_replica.cc:330] T a84b1295cde84ecebfd7625ac7d91ac5 P d22eaf889db749c8b6f5469f69fa6379: stopping tablet replica
I20260620 19:13:59.294153 28975 tablet_replica.cc:330] T a84b1295cde84ecebfd7625ac7d91ac5 P 555af0dc008a4472af78305269d8c363: stopping tablet replica
I20260620 19:13:59.294476 28975 raft_consensus.cc:2229] T a84b1295cde84ecebfd7625ac7d91ac5 P 555af0dc008a4472af78305269d8c363 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:13:59.294512 28977 raft_consensus.cc:2229] T a84b1295cde84ecebfd7625ac7d91ac5 P d22eaf889db749c8b6f5469f69fa6379 [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:13:59.294796 28975 raft_consensus.cc:2258] T a84b1295cde84ecebfd7625ac7d91ac5 P 555af0dc008a4472af78305269d8c363 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:13:59.294884 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 28545
I20260620 19:13:59.294914 28977 raft_consensus.cc:2258] T a84b1295cde84ecebfd7625ac7d91ac5 P d22eaf889db749c8b6f5469f69fa6379 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:13:59.295181 28976 tablet_replica.cc:330] T a84b1295cde84ecebfd7625ac7d91ac5 P 4a12eae723184d198ee0780a913b3aef: stopping tablet replica
I20260620 19:13:59.295460 28976 raft_consensus.cc:2229] T a84b1295cde84ecebfd7625ac7d91ac5 P 4a12eae723184d198ee0780a913b3aef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:13:59.295755 28976 raft_consensus.cc:2258] T a84b1295cde84ecebfd7625ac7d91ac5 P 4a12eae723184d198ee0780a913b3aef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:13:59.296650 28977 ts_tablet_manager.cc:1905] T a84b1295cde84ecebfd7625ac7d91ac5 P d22eaf889db749c8b6f5469f69fa6379: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:13:59.299446 28977 ts_tablet_manager.cc:1918] T a84b1295cde84ecebfd7625ac7d91ac5 P d22eaf889db749c8b6f5469f69fa6379: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:13:59.299546 28977 log.cc:1196] T a84b1295cde84ecebfd7625ac7d91ac5 P d22eaf889db749c8b6f5469f69fa6379: Deleting WAL directory at /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982595231230-9837-0/minicluster-data/ts-2/wal/wals/a84b1295cde84ecebfd7625ac7d91ac5
I20260620 19:13:59.299573 28976 ts_tablet_manager.cc:1905] T a84b1295cde84ecebfd7625ac7d91ac5 P 4a12eae723184d198ee0780a913b3aef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:13:59.299845 28977 ts_tablet_manager.cc:1939] T a84b1295cde84ecebfd7625ac7d91ac5 P d22eaf889db749c8b6f5469f69fa6379: Deleting consensus metadata
I20260620 19:13:59.301061 28471 catalog_manager.cc:4841] TS d22eaf889db749c8b6f5469f69fa6379 (127.9.155.67:34545): tablet a84b1295cde84ecebfd7625ac7d91ac5 (table test-table [id=b42242ad460f472aafd4936f2d115b1a]) successfully deleted
I20260620 19:13:59.301913 28976 ts_tablet_manager.cc:1918] T a84b1295cde84ecebfd7625ac7d91ac5 P 4a12eae723184d198ee0780a913b3aef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:13:59.301999 28976 log.cc:1196] T a84b1295cde84ecebfd7625ac7d91ac5 P 4a12eae723184d198ee0780a913b3aef: Deleting WAL directory at /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982595231230-9837-0/minicluster-data/ts-1/wal/wals/a84b1295cde84ecebfd7625ac7d91ac5
I20260620 19:13:59.302249 28976 ts_tablet_manager.cc:1939] T a84b1295cde84ecebfd7625ac7d91ac5 P 4a12eae723184d198ee0780a913b3aef: Deleting consensus metadata
W20260620 19:13:59.302637 28470 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.155.65:34581 (error 108)
I20260620 19:13:59.302734 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 28676
I20260620 19:13:59.304534 28949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:59.304247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.65:34581 (local address 127.0.0.1:51390)
0620 19:13:59.304399 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:59.304460 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.65:34581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260620 19:13:59.304771 28470 catalog_manager.cc:4584] TS 555af0dc008a4472af78305269d8c363 (127.9.155.65:34581): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a84b1295cde84ecebfd7625ac7d91ac5: Network error: Client connection negotiation failed: client connection to 127.9.155.65:34581: connect: Connection refused (error 111)
I20260620 19:13:59.309386 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 28807
I20260620 19:13:59.309599 28949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:59.309284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.66:40603 (local address 127.0.0.1:51736)
0620 19:13:59.309437 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:59.309507 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.66:40603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 19:13:59.314399 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 28454
2026-06-20T19:13:59Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1071 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:13:59 dist-test-slave-n3fx krb5kdc[28981](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:13:59 dist-test-slave-n3fx krb5kdc[28981](info): set up 2 sockets
Jun 20 19:13:59 dist-test-slave-n3fx krb5kdc[28981](info): commencing operation
krb5kdc: starting...
W20260620 19:14:00.283983 9837 mini_kdc.cc:121] Time spent starting KDC: real 0.950s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/krb5kdc/test-user.keytab.
Jun 20 19:14:00 dist-test-slave-n3fx krb5kdc[28981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:14:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:14:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.155.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.155.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.155.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab.
Entry for principal kudu/127.9.155.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab.
I20260620 19:14:00.406798 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:36513
--webserver_interface=127.9.155.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab
--principal=kudu/127.9.155.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:40837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.155.126:36513
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:14:00.519902 28994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:14:00.520241 28994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:14:00.520305 28994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:14:00.524636 28994 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:14:00.524715 28994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:14:00.524741 28994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:14:00.524766 28994 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:14:00.524796 28994 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:14:00.530503 28994 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:40837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.155.126:36513
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab
--principal=kudu/127.9.155.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:36513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.155.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28994
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:14:00.531679 28994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:14:00.532819 28994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:14:00.541687 29002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:14:00.541718 29000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:14:00.541740 28999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:14:00.542131 28994 server_base.cc:1034] running on GCE node
I20260620 19:14:00.542704 28994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:14:00.543872 28994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:14:00.545025 28994 hybrid_clock.cc:648] HybridClock initialized: now 1781982840544989 us; error 58 us; skew 500 ppm
Jun 20 19:14:00 dist-test-slave-n3fx krb5kdc[28981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982840, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:14:00.547910 28994 init.cc:377] Logged in from keytab as kudu/127.9.155.126@KRBTEST.COM (short username kudu)
I20260620 19:14:00.548913 28994 webserver.cc:460] Webserver started at http://127.9.155.126:39017/ using document root <none> and password file <none>
I20260620 19:14:00.549625 28994 fs_manager.cc:359] Metadata directory not provided
I20260620 19:14:00.549701 28994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:14:00.549902 28994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:14:00.551975 28994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/minicluster-data/master-0/data/instance:
uuid: "31548f16d78f4af1a70ec7a47dad2e9a"
format_stamp: "Formatted at 2026-06-20 19:14:00 on dist-test-slave-n3fx"
server_key: "b485d5689648e3bedbac45bbc31c76ef"
server_key_iv: "cbb2c473afdd9e6d9a54cbd8155beb42"
server_key_version: "encryptionkey@0"
I20260620 19:14:00.552564 28994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/minicluster-data/master-0/wal/instance:
uuid: "31548f16d78f4af1a70ec7a47dad2e9a"
format_stamp: "Formatted at 2026-06-20 19:14:00 on dist-test-slave-n3fx"
server_key: "b485d5689648e3bedbac45bbc31c76ef"
server_key_iv: "cbb2c473afdd9e6d9a54cbd8155beb42"
server_key_version: "encryptionkey@0"
I20260620 19:14:00.556488 28994 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 19:14:00.559276 29009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:00.560775 28994 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:14:00.560904 28994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/minicluster-data/master-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/minicluster-data/master-0/wal
uuid: "31548f16d78f4af1a70ec7a47dad2e9a"
format_stamp: "Formatted at 2026-06-20 19:14:00 on dist-test-slave-n3fx"
server_key: "b485d5689648e3bedbac45bbc31c76ef"
server_key_iv: "cbb2c473afdd9e6d9a54cbd8155beb42"
server_key_version: "encryptionkey@0"
I20260620 19:14:00.561015 28994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:14:00.577924 28994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:14:00.581662 28994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:14:00.581867 28994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:14:00.590078 28994 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.126:36513
I20260620 19:14:00.590097 29061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.126:36513 every 8 connection(s)
I20260620 19:14:00.591075 28994 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
I20260620 19:14:00.593217 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 28994
I20260620 19:14:00.593353 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/minicluster-data/master-0/wal/instance
I20260620 19:14:00.593608 9837 external_mini_cluster.cc:1428] Setting key 9eafff42bc62c994f1866f91e9365cc5
I20260620 19:14:00.594031 29062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:14:00.600063 29062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31548f16d78f4af1a70ec7a47dad2e9a: Bootstrap starting.
Jun 20 19:14:00 dist-test-slave-n3fx krb5kdc[28981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:14:00.602476 29062 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31548f16d78f4af1a70ec7a47dad2e9a: Neither blocks nor log segments found. Creating new log.
I20260620 19:14:00.603166 29062 log.cc:824] T 00000000000000000000000000000000 P 31548f16d78f4af1a70ec7a47dad2e9a: Log is configured to *not* fsync() on all Append() calls
I20260620 19:14:00.605063 29062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31548f16d78f4af1a70ec7a47dad2e9a: No bootstrap required, opened a new log
I20260620 19:14:00.607666 29065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:00.594984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57314 (local address 127.9.155.126:36513)
0620 19:14:00.595441 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:00.595455 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:00.595491 (+ 36us) server_negotiation.cc:407] Connection header received
0620 19:14:00.596196 (+ 705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:00.596240 (+ 44us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:00.596577 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:00.596877 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:00.597665 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:00.598659 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:00.599504 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:00.599969 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:00.600127 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:00.602712 (+ 2585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:00.602718 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:00.602731 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:14:00.602760 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:00.604410 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:00.604826 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:00.604831 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:00.604835 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:00.604953 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:00.605236 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:00.605240 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:00.605242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:00.605779 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:00.605978 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:00.606267 (+ 289us) server_negotiation.cc:299] Negotiation successful
0620 19:14:00.606599 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":141,"threads_started":1}
I20260620 19:14:00.609638 29062 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31548f16d78f4af1a70ec7a47dad2e9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31548f16d78f4af1a70ec7a47dad2e9a" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 36513 } }
I20260620 19:14:00.609887 29062 raft_consensus.cc:374] T 00000000000000000000000000000000 P 31548f16d78f4af1a70ec7a47dad2e9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:14:00.609951 29062 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31548f16d78f4af1a70ec7a47dad2e9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31548f16d78f4af1a70ec7a47dad2e9a, State: Initialized, Role: FOLLOWER
I20260620 19:14:00.610385 29062 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31548f16d78f4af1a70ec7a47dad2e9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31548f16d78f4af1a70ec7a47dad2e9a" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 36513 } }
I20260620 19:14:00.610524 29062 raft_consensus.cc:388] T 00000000000000000000000000000000 P 31548f16d78f4af1a70ec7a47dad2e9a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:14:00.610571 29062 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31548f16d78f4af1a70ec7a47dad2e9a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:14:00.610644 29062 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31548f16d78f4af1a70ec7a47dad2e9a [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:14:00.611545 29062 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31548f16d78f4af1a70ec7a47dad2e9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31548f16d78f4af1a70ec7a47dad2e9a" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 36513 } }
I20260620 19:14:00.611855 29062 leader_election.cc:302] T 00000000000000000000000000000000 P 31548f16d78f4af1a70ec7a47dad2e9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31548f16d78f4af1a70ec7a47dad2e9a; no voters:
I20260620 19:14:00.612128 29062 leader_election.cc:288] T 00000000000000000000000000000000 P 31548f16d78f4af1a70ec7a47dad2e9a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:14:00.612331 29067 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31548f16d78f4af1a70ec7a47dad2e9a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:14:00.612534 29067 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31548f16d78f4af1a70ec7a47dad2e9a [term 1 LEADER]: Becoming Leader. State: Replica: 31548f16d78f4af1a70ec7a47dad2e9a, State: Running, Role: LEADER
I20260620 19:14:00.612854 29067 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31548f16d78f4af1a70ec7a47dad2e9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31548f16d78f4af1a70ec7a47dad2e9a" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 36513 } }
I20260620 19:14:00.613338 29062 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31548f16d78f4af1a70ec7a47dad2e9a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:14:00.614534 29068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31548f16d78f4af1a70ec7a47dad2e9a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "31548f16d78f4af1a70ec7a47dad2e9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31548f16d78f4af1a70ec7a47dad2e9a" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 36513 } } }
I20260620 19:14:00.614655 29068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31548f16d78f4af1a70ec7a47dad2e9a [sys.catalog]: This master's current role is: LEADER
I20260620 19:14:00.614949 29069 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31548f16d78f4af1a70ec7a47dad2e9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31548f16d78f4af1a70ec7a47dad2e9a. Latest consensus state: current_term: 1 leader_uuid: "31548f16d78f4af1a70ec7a47dad2e9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31548f16d78f4af1a70ec7a47dad2e9a" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 36513 } } }
I20260620 19:14:00.615036 29069 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31548f16d78f4af1a70ec7a47dad2e9a [sys.catalog]: This master's current role is: LEADER
I20260620 19:14:00.615139 29076 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:14:00.618412 29076 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:14:00.624378 29076 catalog_manager.cc:1347] Generated new cluster ID: e2a005c2acdb4957bbe277776e0f0a15
I20260620 19:14:00.624456 29076 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:14:00.634497 29076 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:14:00.635869 29076 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:14:00.641027 29076 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 31548f16d78f4af1a70ec7a47dad2e9a: Generated new TSK 0
I20260620 19:14:00.641851 29076 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.155.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.155.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.155.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.65.keytab.
Entry for principal kudu/127.9.155.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.65.keytab.
I20260620 19:14:00.671303 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.65:0
--local_ip_for_outbound_sockets=127.9.155.65
--webserver_interface=127.9.155.65
--webserver_port=0
--tserver_master_addrs=127.9.155.126:36513
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.65.keytab
--principal=kudu/127.9.155.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:40837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:14:00.783854 29088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:14:00.784140 29088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:14:00.784186 29088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:14:00.788363 29088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:14:00.788424 29088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:14:00.788487 29088 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.65
I20260620 19:14:00.793946 29088 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:40837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.65.keytab
--principal=kudu/127.9.155.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.155.65
--webserver_port=0
--tserver_master_addrs=127.9.155.126:36513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.65
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:14:00.795023 29088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:14:00.796012 29088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:14:00.805704 29093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:14:00.805822 29096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:14:00.805794 29094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:14:00.806265 29088 server_base.cc:1034] running on GCE node
I20260620 19:14:00.806664 29088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:14:00.807310 29088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:14:00.808461 29088 hybrid_clock.cc:648] HybridClock initialized: now 1781982840808436 us; error 39 us; skew 500 ppm
Jun 20 19:14:00 dist-test-slave-n3fx krb5kdc[28981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982840, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:14:00.811115 29088 init.cc:377] Logged in from keytab as kudu/127.9.155.65@KRBTEST.COM (short username kudu)
I20260620 19:14:00.811976 29088 webserver.cc:460] Webserver started at http://127.9.155.65:45929/ using document root <none> and password file <none>
I20260620 19:14:00.812551 29088 fs_manager.cc:359] Metadata directory not provided
I20260620 19:14:00.812613 29088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:14:00.812848 29088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:14:00.814996 29088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/minicluster-data/ts-0/data/instance:
uuid: "eb0540bcdcf840fc827256983a122092"
format_stamp: "Formatted at 2026-06-20 19:14:00 on dist-test-slave-n3fx"
server_key: "fbf510d8b55574ac5e6b9192b336c559"
server_key_iv: "de15a834cec2cb1d65afd1b27e44f950"
server_key_version: "encryptionkey@0"
I20260620 19:14:00.815523 29088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/minicluster-data/ts-0/wal/instance:
uuid: "eb0540bcdcf840fc827256983a122092"
format_stamp: "Formatted at 2026-06-20 19:14:00 on dist-test-slave-n3fx"
server_key: "fbf510d8b55574ac5e6b9192b336c559"
server_key_iv: "de15a834cec2cb1d65afd1b27e44f950"
server_key_version: "encryptionkey@0"
I20260620 19:14:00.819254 29088 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:14:00.821933 29103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:00.823326 29088 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 19:14:00.823441 29088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/minicluster-data/ts-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/minicluster-data/ts-0/wal
uuid: "eb0540bcdcf840fc827256983a122092"
format_stamp: "Formatted at 2026-06-20 19:14:00 on dist-test-slave-n3fx"
server_key: "fbf510d8b55574ac5e6b9192b336c559"
server_key_iv: "de15a834cec2cb1d65afd1b27e44f950"
server_key_version: "encryptionkey@0"
I20260620 19:14:00.823558 29088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:14:00.847047 29088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:14:00.850411 29088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:14:00.850582 29088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:14:00.851193 29088 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:14:00.852023 29088 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:14:00.852069 29088 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:00.852109 29088 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:14:00.852136 29088 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:00.860630 29088 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.65:37971
I20260620 19:14:00.860707 29216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.65:37971 every 8 connection(s)
I20260620 19:14:00.861524 29088 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
I20260620 19:14:00.867846 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 29088
I20260620 19:14:00.867977 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/minicluster-data/ts-0/wal/instance
I20260620 19:14:00.868297 9837 external_mini_cluster.cc:1428] Setting key d1df3af29f7f5e867441bbb8991cef73
Jun 20 19:14:00 dist-test-slave-n3fx krb5kdc[28981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982840, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.65@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:14:00.876190 29065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:00.863373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.65:46079 (local address 127.9.155.126:36513)
0620 19:14:00.863518 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:00.863522 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:00.864411 (+ 889us) server_negotiation.cc:407] Connection header received
0620 19:14:00.865354 (+ 943us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:00.865358 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:00.865415 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:00.865486 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:00.866718 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:00.867263 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:00.868521 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:00.868758 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:00.868851 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:00.871914 (+ 3063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:00.871919 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:00.871922 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:14:00.871954 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:00.873515 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:00.874399 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:00.874403 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:00.874404 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:00.874495 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:00.874926 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:00.874929 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:00.874930 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:00.875131 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:00.875210 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:00.875915 (+ 705us) server_negotiation.cc:299] Negotiation successful
0620 19:14:00.876058 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 19:14:00.877391 29219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:00.863753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:36513 (local address 127.9.155.65:46079)
0620 19:14:00.864255 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:00.864315 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 19:14:00.865135 (+ 820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:14:00.865711 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:14:00.865718 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:14:00.866082 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:14:00.866566 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:00.866581 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:00.867387 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:00.867391 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:00.868405 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:00.868416 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:00.868906 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:00.868910 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:00.869092 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:00.869942 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:14:00.869964 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:14:00.871710 (+ 1746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:14:00.873813 (+ 2103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:00.873820 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:00.873840 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:00.874232 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:00.874603 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:00.874607 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:00.874609 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:00.874832 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:00.875232 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:14:00.875239 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:14:00.875605 (+ 366us) client_negotiation.cc:769] Sending connection context
0620 19:14:00.875844 (+ 239us) client_negotiation.cc:240] Negotiation successful
0620 19:14:00.876259 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":112,"threads_started":1}
WARNING: no policy specified for kudu/127.9.155.66@KRBTEST.COM; defaulting to no policy
I20260620 19:14:00.878718 29217 heartbeater.cc:347] Connected to a master server at 127.9.155.126:36513
I20260620 19:14:00.878953 29217 heartbeater.cc:464] Registering TS with master...
I20260620 19:14:00.879448 29217 heartbeater.cc:511] Master 127.9.155.126:36513 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.155.66@KRBTEST.COM" created.
I20260620 19:14:00.881157 29026 ts_manager.cc:194] Registered new tserver with Master: eb0540bcdcf840fc827256983a122092 (127.9.155.65:37971)
I20260620 19:14:00.882395 29026 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.155.65@KRBTEST.COM'} at 127.9.155.65:46079
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.155.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.66.keytab.
Entry for principal kudu/127.9.155.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.66.keytab.
I20260620 19:14:00.900329 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.66:0
--local_ip_for_outbound_sockets=127.9.155.66
--webserver_interface=127.9.155.66
--webserver_port=0
--tserver_master_addrs=127.9.155.126:36513
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.66.keytab
--principal=kudu/127.9.155.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:40837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:14:01.011391 29222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:14:01.011724 29222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:14:01.011772 29222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:14:01.016148 29222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:14:01.016251 29222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:14:01.016323 29222 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.66
I20260620 19:14:01.021826 29222 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:40837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.66.keytab
--principal=kudu/127.9.155.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.155.66
--webserver_port=0
--tserver_master_addrs=127.9.155.126:36513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29222
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.66
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:14:01.022893 29222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:14:01.023875 29222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:14:01.033284 29227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:14:01.033330 29222 server_base.cc:1034] running on GCE node
W20260620 19:14:01.033284 29230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:14:01.033303 29228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:14:01.034003 29222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:14:01.034622 29222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:14:01.035794 29222 hybrid_clock.cc:648] HybridClock initialized: now 1781982841035770 us; error 33 us; skew 500 ppm
Jun 20 19:14:01 dist-test-slave-n3fx krb5kdc[28981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982841, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:14:01.038434 29222 init.cc:377] Logged in from keytab as kudu/127.9.155.66@KRBTEST.COM (short username kudu)
I20260620 19:14:01.039328 29222 webserver.cc:460] Webserver started at http://127.9.155.66:33671/ using document root <none> and password file <none>
I20260620 19:14:01.039888 29222 fs_manager.cc:359] Metadata directory not provided
I20260620 19:14:01.039951 29222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:14:01.040139 29222 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:14:01.042237 29222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/minicluster-data/ts-1/data/instance:
uuid: "337d18064754461e9de8897a82635eee"
format_stamp: "Formatted at 2026-06-20 19:14:01 on dist-test-slave-n3fx"
server_key: "35aa8d0dc5e3b2213e5e559925f6f6e2"
server_key_iv: "60a3f7e222f9c30ae3f7ffd7d72ec030"
server_key_version: "encryptionkey@0"
I20260620 19:14:01.042793 29222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/minicluster-data/ts-1/wal/instance:
uuid: "337d18064754461e9de8897a82635eee"
format_stamp: "Formatted at 2026-06-20 19:14:01 on dist-test-slave-n3fx"
server_key: "35aa8d0dc5e3b2213e5e559925f6f6e2"
server_key_iv: "60a3f7e222f9c30ae3f7ffd7d72ec030"
server_key_version: "encryptionkey@0"
I20260620 19:14:01.046427 29222 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 19:14:01.049002 29237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:01.050276 29222 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:14:01.050374 29222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/minicluster-data/ts-1/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/minicluster-data/ts-1/wal
uuid: "337d18064754461e9de8897a82635eee"
format_stamp: "Formatted at 2026-06-20 19:14:01 on dist-test-slave-n3fx"
server_key: "35aa8d0dc5e3b2213e5e559925f6f6e2"
server_key_iv: "60a3f7e222f9c30ae3f7ffd7d72ec030"
server_key_version: "encryptionkey@0"
I20260620 19:14:01.050477 29222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:14:01.066817 29222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:14:01.070158 29222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:14:01.070336 29222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:14:01.070940 29222 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:14:01.071755 29222 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:14:01.071802 29222 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:01.071842 29222 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:14:01.071861 29222 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:01.080134 29222 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.66:46401
I20260620 19:14:01.080178 29350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.66:46401 every 8 connection(s)
I20260620 19:14:01.081027 29222 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
I20260620 19:14:01.086838 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 29222
I20260620 19:14:01.086975 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/minicluster-data/ts-1/wal/instance
I20260620 19:14:01.087302 9837 external_mini_cluster.cc:1428] Setting key 1f80a727efc9980b14747fb30fdcdcc8
Jun 20 19:14:01 dist-test-slave-n3fx krb5kdc[28981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982841, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.66@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:14:01.095466 29065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:01.082755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.66:58831 (local address 127.9.155.126:36513)
0620 19:14:01.082876 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:01.082880 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:01.083684 (+ 804us) server_negotiation.cc:407] Connection header received
0620 19:14:01.084625 (+ 941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:01.084628 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:01.084677 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:01.084749 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:01.085986 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:01.086575 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:01.088249 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:01.088568 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:01.088676 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:01.091562 (+ 2886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:01.091566 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:01.091568 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:14:01.091595 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:01.093067 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:01.093763 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:01.093766 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:01.093768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:01.093856 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:01.094292 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:01.094295 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:01.094296 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:01.094492 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:01.094573 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:01.095231 (+ 658us) server_negotiation.cc:299] Negotiation successful
0620 19:14:01.095340 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 19:14:01.096786 29353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:01.083071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:36513 (local address 127.9.155.66:58831)
0620 19:14:01.083521 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:01.083563 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 19:14:01.084412 (+ 849us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:14:01.084918 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:14:01.084925 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:14:01.085328 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:14:01.085810 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:01.085824 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:01.086726 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:01.086730 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:01.088092 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:01.088103 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:01.088752 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:01.088757 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:01.088942 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:01.089615 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:14:01.089637 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:14:01.091358 (+ 1721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:14:01.093188 (+ 1830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:01.093194 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:01.093214 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:01.093618 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:01.093952 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:01.093955 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:01.093957 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:01.094172 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:01.094595 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:14:01.094602 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:14:01.094963 (+ 361us) client_negotiation.cc:769] Sending connection context
0620 19:14:01.095188 (+ 225us) client_negotiation.cc:240] Negotiation successful
0620 19:14:01.095650 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":105,"threads_started":1}
WARNING: no policy specified for kudu/127.9.155.67@KRBTEST.COM; defaulting to no policy
I20260620 19:14:01.097949 29351 heartbeater.cc:347] Connected to a master server at 127.9.155.126:36513
I20260620 19:14:01.098173 29351 heartbeater.cc:464] Registering TS with master...
I20260620 19:14:01.098667 29351 heartbeater.cc:511] Master 127.9.155.126:36513 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.155.67@KRBTEST.COM" created.
I20260620 19:14:01.099790 29026 ts_manager.cc:194] Registered new tserver with Master: 337d18064754461e9de8897a82635eee (127.9.155.66:46401)
I20260620 19:14:01.100433 29026 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.155.66@KRBTEST.COM'} at 127.9.155.66:58831
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.155.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.67.keytab.
Entry for principal kudu/127.9.155.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.67.keytab.
I20260620 19:14:01.116026 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.67:0
--local_ip_for_outbound_sockets=127.9.155.67
--webserver_interface=127.9.155.67
--webserver_port=0
--tserver_master_addrs=127.9.155.126:36513
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.67.keytab
--principal=kudu/127.9.155.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:40837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:14:01.225075 29356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:14:01.225344 29356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:14:01.225387 29356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:14:01.229513 29356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:14:01.229583 29356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:14:01.229655 29356 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.67
I20260620 19:14:01.236018 29356 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:40837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.67.keytab
--principal=kudu/127.9.155.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.155.67
--webserver_port=0
--tserver_master_addrs=127.9.155.126:36513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29356
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.67
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:14:01.237178 29356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:14:01.238216 29356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:14:01.247676 29362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:14:01.247816 29356 server_base.cc:1034] running on GCE node
W20260620 19:14:01.247684 29361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:14:01.247690 29364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:14:01.248459 29356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:14:01.249089 29356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:14:01.250248 29356 hybrid_clock.cc:648] HybridClock initialized: now 1781982841250222 us; error 40 us; skew 500 ppm
Jun 20 19:14:01 dist-test-slave-n3fx krb5kdc[28981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982841, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:14:01.252921 29356 init.cc:377] Logged in from keytab as kudu/127.9.155.67@KRBTEST.COM (short username kudu)
I20260620 19:14:01.253830 29356 webserver.cc:460] Webserver started at http://127.9.155.67:36481/ using document root <none> and password file <none>
I20260620 19:14:01.254395 29356 fs_manager.cc:359] Metadata directory not provided
I20260620 19:14:01.254460 29356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:14:01.254668 29356 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:14:01.256767 29356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/minicluster-data/ts-2/data/instance:
uuid: "97260049ae6b4537892ea05bbcc6af7d"
format_stamp: "Formatted at 2026-06-20 19:14:01 on dist-test-slave-n3fx"
server_key: "7e7298a97c55d015b1f40abe92a027f9"
server_key_iv: "40d00b9acd6413bd29b10fd0344cfbd6"
server_key_version: "encryptionkey@0"
I20260620 19:14:01.257287 29356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/minicluster-data/ts-2/wal/instance:
uuid: "97260049ae6b4537892ea05bbcc6af7d"
format_stamp: "Formatted at 2026-06-20 19:14:01 on dist-test-slave-n3fx"
server_key: "7e7298a97c55d015b1f40abe92a027f9"
server_key_iv: "40d00b9acd6413bd29b10fd0344cfbd6"
server_key_version: "encryptionkey@0"
I20260620 19:14:01.260844 29356 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 19:14:01.263312 29371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:01.264642 29356 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 19:14:01.264747 29356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/minicluster-data/ts-2/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/minicluster-data/ts-2/wal
uuid: "97260049ae6b4537892ea05bbcc6af7d"
format_stamp: "Formatted at 2026-06-20 19:14:01 on dist-test-slave-n3fx"
server_key: "7e7298a97c55d015b1f40abe92a027f9"
server_key_iv: "40d00b9acd6413bd29b10fd0344cfbd6"
server_key_version: "encryptionkey@0"
I20260620 19:14:01.264851 29356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:14:01.284132 29356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:14:01.287352 29356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:14:01.287535 29356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:14:01.288146 29356 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:14:01.289060 29356 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:14:01.289110 29356 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:01.289175 29356 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:14:01.289201 29356 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:01.298053 29356 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.67:34359
I20260620 19:14:01.298089 29484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.67:34359 every 8 connection(s)
I20260620 19:14:01.298964 29356 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
I20260620 19:14:01.302474 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 29356
I20260620 19:14:01.302606 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/minicluster-data/ts-2/wal/instance
I20260620 19:14:01.302937 9837 external_mini_cluster.cc:1428] Setting key 5458b283567ffa3f9bde2094b88a0dd3
Jun 20 19:14:01 dist-test-slave-n3fx krb5kdc[28981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982841, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.67@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:14:01.311740 29065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:01.300779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.67:54307 (local address 127.9.155.126:36513)
0620 19:14:01.300911 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:01.300915 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:01.301733 (+ 818us) server_negotiation.cc:407] Connection header received
0620 19:14:01.302854 (+ 1121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:01.302857 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:01.302906 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:01.302986 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:01.304301 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:01.304823 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:01.305940 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:01.306163 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:01.306249 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:01.308593 (+ 2344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:01.308596 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:01.308599 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:14:01.308622 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:01.309891 (+ 1269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:01.310358 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:01.310361 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:01.310363 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:01.310435 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:01.310737 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:01.310740 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:01.310742 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:01.310916 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:01.310997 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:01.311459 (+ 462us) server_negotiation.cc:299] Negotiation successful
0620 19:14:01.311580 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 19:14:01.312552 29487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:01.301118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:36513 (local address 127.9.155.67:54307)
0620 19:14:01.301566 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:01.301626 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 19:14:01.302591 (+ 965us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:14:01.303184 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:14:01.303193 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:14:01.303566 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:14:01.304156 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:01.304172 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:01.304937 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:01.304940 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:01.305826 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:01.305835 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:01.306295 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:01.306299 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:01.306452 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:01.306972 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:14:01.307000 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:14:01.308422 (+ 1422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:14:01.309994 (+ 1572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:01.309999 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:01.310014 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:01.310267 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:01.310513 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:01.310516 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:01.310518 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:01.310655 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:01.311009 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:14:01.311015 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:14:01.311246 (+ 231us) client_negotiation.cc:769] Sending connection context
0620 19:14:01.311406 (+ 160us) client_negotiation.cc:240] Negotiation successful
0620 19:14:01.311719 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":109,"threads_started":1}
I20260620 19:14:01.313635 29485 heartbeater.cc:347] Connected to a master server at 127.9.155.126:36513
I20260620 19:14:01.313828 29485 heartbeater.cc:464] Registering TS with master...
I20260620 19:14:01.314301 29485 heartbeater.cc:511] Master 127.9.155.126:36513 requested a full tablet report, sending...
I20260620 19:14:01.315292 29026 ts_manager.cc:194] Registered new tserver with Master: 97260049ae6b4537892ea05bbcc6af7d (127.9.155.67:34359)
I20260620 19:14:01.315861 29026 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.155.67@KRBTEST.COM'} at 127.9.155.67:54307
I20260620 19:14:01.316686 9837 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:14:01.325438 29065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:01.318621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57326 (local address 127.9.155.126:36513)
0620 19:14:01.318775 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:01.318780 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:01.318898 (+ 118us) server_negotiation.cc:407] Connection header received
0620 19:14:01.319017 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:01.319020 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:01.319062 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:01.319125 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:01.319792 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:01.320370 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:01.321218 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:01.321420 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:01.321493 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:01.322784 (+ 1291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:01.322787 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:01.322789 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:14:01.322810 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:01.324062 (+ 1252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:01.324471 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:01.324473 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:01.324474 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:01.324547 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:01.324735 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:01.324737 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:01.324738 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:01.324930 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:01.325024 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:01.325211 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 19:14:01.325296 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 19:14:01.328313 29026 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57326:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:14:01.331589 29026 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:14:01.346151 29500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:01.340866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35346 (local address 127.9.155.65:37971)
0620 19:14:01.341554 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:01.341568 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:01.341588 (+ 20us) server_negotiation.cc:407] Connection header received
0620 19:14:01.341657 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:01.341662 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:01.341827 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:01.341972 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:14:01.342166 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:01.343245 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:01.344969 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:01.345508 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:01.345610 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:01.345773 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:01.345900 (+ 127us) server_negotiation.cc:299] Negotiation successful
0620 19:14:01.345991 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":129,"threads_started":1}
I20260620 19:14:01.346141 29499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:01.340266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.65:37971 (local address 127.0.0.1:35346)
0620 19:14:01.341254 (+ 988us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:01.341266 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:14:01.341346 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:14:01.341954 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:14:01.341957 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:14:01.341972 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:14:01.342078 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:01.342085 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:01.343347 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:01.343350 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:01.344835 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:01.344845 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:01.345625 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:01.345629 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:01.345736 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:01.345772 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 19:14:01.345874 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 19:14:01.345991 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":927,"mutex_wait_us":92,"thread_start_us":89,"threads_started":1}
I20260620 19:14:01.346309 29498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:01.340035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.66:46401 (local address 127.0.0.1:40526)
0620 19:14:01.340740 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:01.340793 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 19:14:01.340926 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:14:01.341696 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:14:01.341702 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:14:01.341723 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:14:01.341862 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:01.341869 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:01.343397 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:01.343400 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:01.344650 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:01.344659 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:01.345872 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:01.345876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:01.345964 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:01.346022 (+ 58us) client_negotiation.cc:769] Sending connection context
0620 19:14:01.346096 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 19:14:01.346163 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"mutex_wait_us":78,"thread_start_us":86,"threads_started":1}
I20260620 19:14:01.346645 29497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:01.340188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40526 (local address 127.9.155.66:46401)
0620 19:14:01.340550 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:01.340562 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:01.341063 (+ 501us) server_negotiation.cc:407] Connection header received
0620 19:14:01.341143 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:01.341149 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:01.341356 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:01.341509 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:14:01.341983 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:01.343226 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:01.344765 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:01.345566 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:01.345671 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:01.345809 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:01.346411 (+ 602us) server_negotiation.cc:299] Negotiation successful
0620 19:14:01.346509 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":101,"threads_started":1}
I20260620 19:14:01.346776 29501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:01.340936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45286 (local address 127.9.155.67:34359)
0620 19:14:01.341709 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:01.341721 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:01.341742 (+ 21us) server_negotiation.cc:407] Connection header received
0620 19:14:01.341803 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:01.341806 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:01.341947 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:01.342079 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:14:01.342402 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:01.343146 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:01.344957 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:01.345731 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:01.345821 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:01.345932 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:01.346554 (+ 622us) server_negotiation.cc:299] Negotiation successful
0620 19:14:01.346644 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":79,"threads_started":1}
I20260620 19:14:01.346998 29496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:01.340779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.67:34359 (local address 127.0.0.1:45286)
0620 19:14:01.341336 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:01.341350 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:14:01.341423 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:14:01.342229 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:14:01.342231 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:14:01.342245 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:14:01.342304 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:01.342308 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:01.343263 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:01.343266 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:01.344835 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:01.344845 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:01.345983 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:01.345986 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:01.346093 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:01.346113 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 19:14:01.346165 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 19:14:01.346218 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"mutex_wait_us":176,"thread_start_us":88,"threads_started":1}
I20260620 19:14:01.349462 29285 tablet_service.cc:1467] Processing CreateTablet for tablet 587330cb1729499b986887e0149fdd13 (DEFAULT_TABLE table=test-table [id=d8a82dbdd183443bba79670450ebcd9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:14:01.349496 29419 tablet_service.cc:1467] Processing CreateTablet for tablet 587330cb1729499b986887e0149fdd13 (DEFAULT_TABLE table=test-table [id=d8a82dbdd183443bba79670450ebcd9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:14:01.349498 29151 tablet_service.cc:1467] Processing CreateTablet for tablet 587330cb1729499b986887e0149fdd13 (DEFAULT_TABLE table=test-table [id=d8a82dbdd183443bba79670450ebcd9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:14:01.350446 29285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 587330cb1729499b986887e0149fdd13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:14:01.350605 29151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 587330cb1729499b986887e0149fdd13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:14:01.350644 29419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 587330cb1729499b986887e0149fdd13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:14:01.356812 29502 tablet_bootstrap.cc:492] T 587330cb1729499b986887e0149fdd13 P 337d18064754461e9de8897a82635eee: Bootstrap starting.
I20260620 19:14:01.358178 29503 tablet_bootstrap.cc:492] T 587330cb1729499b986887e0149fdd13 P 97260049ae6b4537892ea05bbcc6af7d: Bootstrap starting.
I20260620 19:14:01.358215 29504 tablet_bootstrap.cc:492] T 587330cb1729499b986887e0149fdd13 P eb0540bcdcf840fc827256983a122092: Bootstrap starting.
I20260620 19:14:01.359652 29502 tablet_bootstrap.cc:654] T 587330cb1729499b986887e0149fdd13 P 337d18064754461e9de8897a82635eee: Neither blocks nor log segments found. Creating new log.
I20260620 19:14:01.359910 29503 tablet_bootstrap.cc:654] T 587330cb1729499b986887e0149fdd13 P 97260049ae6b4537892ea05bbcc6af7d: Neither blocks nor log segments found. Creating new log.
I20260620 19:14:01.360276 29504 tablet_bootstrap.cc:654] T 587330cb1729499b986887e0149fdd13 P eb0540bcdcf840fc827256983a122092: Neither blocks nor log segments found. Creating new log.
I20260620 19:14:01.360607 29502 log.cc:824] T 587330cb1729499b986887e0149fdd13 P 337d18064754461e9de8897a82635eee: Log is configured to *not* fsync() on all Append() calls
I20260620 19:14:01.360940 29503 log.cc:824] T 587330cb1729499b986887e0149fdd13 P 97260049ae6b4537892ea05bbcc6af7d: Log is configured to *not* fsync() on all Append() calls
I20260620 19:14:01.361372 29504 log.cc:824] T 587330cb1729499b986887e0149fdd13 P eb0540bcdcf840fc827256983a122092: Log is configured to *not* fsync() on all Append() calls
I20260620 19:14:01.362802 29503 tablet_bootstrap.cc:492] T 587330cb1729499b986887e0149fdd13 P 97260049ae6b4537892ea05bbcc6af7d: No bootstrap required, opened a new log
I20260620 19:14:01.362824 29502 tablet_bootstrap.cc:492] T 587330cb1729499b986887e0149fdd13 P 337d18064754461e9de8897a82635eee: No bootstrap required, opened a new log
I20260620 19:14:01.362995 29503 ts_tablet_manager.cc:1397] T 587330cb1729499b986887e0149fdd13 P 97260049ae6b4537892ea05bbcc6af7d: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 19:14:01.363034 29502 ts_tablet_manager.cc:1397] T 587330cb1729499b986887e0149fdd13 P 337d18064754461e9de8897a82635eee: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 19:14:01.363183 29504 tablet_bootstrap.cc:492] T 587330cb1729499b986887e0149fdd13 P eb0540bcdcf840fc827256983a122092: No bootstrap required, opened a new log
I20260620 19:14:01.363358 29504 ts_tablet_manager.cc:1397] T 587330cb1729499b986887e0149fdd13 P eb0540bcdcf840fc827256983a122092: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 19:14:01.366312 29503 raft_consensus.cc:348] T 587330cb1729499b986887e0149fdd13 P 97260049ae6b4537892ea05bbcc6af7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "337d18064754461e9de8897a82635eee" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 46401 } } peers { permanent_uuid: "eb0540bcdcf840fc827256983a122092" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 37971 } } peers { permanent_uuid: "97260049ae6b4537892ea05bbcc6af7d" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34359 } }
I20260620 19:14:01.366571 29503 raft_consensus.cc:374] T 587330cb1729499b986887e0149fdd13 P 97260049ae6b4537892ea05bbcc6af7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:14:01.366629 29503 raft_consensus.cc:729] T 587330cb1729499b986887e0149fdd13 P 97260049ae6b4537892ea05bbcc6af7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97260049ae6b4537892ea05bbcc6af7d, State: Initialized, Role: FOLLOWER
I20260620 19:14:01.367086 29503 consensus_queue.cc:260] T 587330cb1729499b986887e0149fdd13 P 97260049ae6b4537892ea05bbcc6af7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "337d18064754461e9de8897a82635eee" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 46401 } } peers { permanent_uuid: "eb0540bcdcf840fc827256983a122092" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 37971 } } peers { permanent_uuid: "97260049ae6b4537892ea05bbcc6af7d" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34359 } }
I20260620 19:14:01.366981 29504 raft_consensus.cc:348] T 587330cb1729499b986887e0149fdd13 P eb0540bcdcf840fc827256983a122092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "337d18064754461e9de8897a82635eee" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 46401 } } peers { permanent_uuid: "eb0540bcdcf840fc827256983a122092" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 37971 } } peers { permanent_uuid: "97260049ae6b4537892ea05bbcc6af7d" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34359 } }
I20260620 19:14:01.366981 29502 raft_consensus.cc:348] T 587330cb1729499b986887e0149fdd13 P 337d18064754461e9de8897a82635eee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "337d18064754461e9de8897a82635eee" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 46401 } } peers { permanent_uuid: "eb0540bcdcf840fc827256983a122092" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 37971 } } peers { permanent_uuid: "97260049ae6b4537892ea05bbcc6af7d" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34359 } }
I20260620 19:14:01.367256 29504 raft_consensus.cc:374] T 587330cb1729499b986887e0149fdd13 P eb0540bcdcf840fc827256983a122092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:14:01.367266 29502 raft_consensus.cc:374] T 587330cb1729499b986887e0149fdd13 P 337d18064754461e9de8897a82635eee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:14:01.367307 29502 raft_consensus.cc:729] T 587330cb1729499b986887e0149fdd13 P 337d18064754461e9de8897a82635eee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 337d18064754461e9de8897a82635eee, State: Initialized, Role: FOLLOWER
I20260620 19:14:01.367307 29504 raft_consensus.cc:729] T 587330cb1729499b986887e0149fdd13 P eb0540bcdcf840fc827256983a122092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb0540bcdcf840fc827256983a122092, State: Initialized, Role: FOLLOWER
I20260620 19:14:01.367725 29502 consensus_queue.cc:260] T 587330cb1729499b986887e0149fdd13 P 337d18064754461e9de8897a82635eee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "337d18064754461e9de8897a82635eee" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 46401 } } peers { permanent_uuid: "eb0540bcdcf840fc827256983a122092" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 37971 } } peers { permanent_uuid: "97260049ae6b4537892ea05bbcc6af7d" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34359 } }
I20260620 19:14:01.367725 29504 consensus_queue.cc:260] T 587330cb1729499b986887e0149fdd13 P eb0540bcdcf840fc827256983a122092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "337d18064754461e9de8897a82635eee" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 46401 } } peers { permanent_uuid: "eb0540bcdcf840fc827256983a122092" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 37971 } } peers { permanent_uuid: "97260049ae6b4537892ea05bbcc6af7d" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34359 } }
I20260620 19:14:01.368067 29503 ts_tablet_manager.cc:1428] T 587330cb1729499b986887e0149fdd13 P 97260049ae6b4537892ea05bbcc6af7d: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 19:14:01.368489 29485 heartbeater.cc:503] Master 127.9.155.126:36513 was elected leader, sending a full tablet report...
I20260620 19:14:01.368741 29504 ts_tablet_manager.cc:1428] T 587330cb1729499b986887e0149fdd13 P eb0540bcdcf840fc827256983a122092: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 19:14:01.368741 29502 ts_tablet_manager.cc:1428] T 587330cb1729499b986887e0149fdd13 P 337d18064754461e9de8897a82635eee: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 19:14:01.369073 29351 heartbeater.cc:503] Master 127.9.155.126:36513 was elected leader, sending a full tablet report...
I20260620 19:14:01.369410 29217 heartbeater.cc:503] Master 127.9.155.126:36513 was elected leader, sending a full tablet report...
I20260620 19:14:01.419991 29510 raft_consensus.cc:482] T 587330cb1729499b986887e0149fdd13 P 337d18064754461e9de8897a82635eee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:14:01.420197 29510 raft_consensus.cc:504] T 587330cb1729499b986887e0149fdd13 P 337d18064754461e9de8897a82635eee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "337d18064754461e9de8897a82635eee" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 46401 } } peers { permanent_uuid: "eb0540bcdcf840fc827256983a122092" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 37971 } } peers { permanent_uuid: "97260049ae6b4537892ea05bbcc6af7d" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34359 } }
I20260620 19:14:01.421304 29510 leader_election.cc:288] T 587330cb1729499b986887e0149fdd13 P 337d18064754461e9de8897a82635eee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb0540bcdcf840fc827256983a122092 (127.9.155.65:37971), 97260049ae6b4537892ea05bbcc6af7d (127.9.155.67:34359)
I20260620 19:14:01.425011 29501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:01.421699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.66:41917 (local address 127.9.155.67:34359)
0620 19:14:01.421844 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:01.421848 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:01.422018 (+ 170us) server_negotiation.cc:407] Connection header received
0620 19:14:01.422126 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:01.422128 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:01.422182 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:01.422254 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:14:01.422501 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:01.423004 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:01.424040 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:01.424447 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:01.424547 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:01.424595 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:01.424806 (+ 211us) server_negotiation.cc:299] Negotiation successful
0620 19:14:01.424880 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 19:14:01.425012 29353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:01.421524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.65:37971 (local address 127.9.155.66:57565)
0620 19:14:01.421860 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:01.421874 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:14:01.421958 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:14:01.422187 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:14:01.422190 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:14:01.422204 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:14:01.422329 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:01.422336 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:01.423076 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:01.423079 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:01.423895 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:01.423904 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:01.424543 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:01.424546 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:01.424710 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:01.424727 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 19:14:01.424793 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 19:14:01.424883 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"mutex_wait_us":60}
I20260620 19:14:01.425021 29500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:01.421737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.66:57565 (local address 127.9.155.65:37971)
0620 19:14:01.421938 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:01.421942 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:01.421959 (+ 17us) server_negotiation.cc:407] Connection header received
0620 19:14:01.422048 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:01.422052 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:01.422101 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:01.422167 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:14:01.422437 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:01.422962 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:01.424018 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:01.424433 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:01.424536 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:01.424594 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:01.424823 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 19:14:01.424891 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260620 19:14:01.425033 29511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:01.421567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.67:34359 (local address 127.9.155.66:41917)
0620 19:14:01.421924 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:01.421937 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:14:01.422046 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:14:01.422279 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:14:01.422281 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:14:01.422294 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:14:01.422403 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:01.422409 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:01.423109 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:01.423111 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:01.423922 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:01.423930 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:01.424554 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:01.424558 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:01.424683 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:01.424717 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 19:14:01.424777 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 19:14:01.424863 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":44,"thread_start_us":128,"threads_started":1}
I20260620 19:14:01.425637 29171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "587330cb1729499b986887e0149fdd13" candidate_uuid: "337d18064754461e9de8897a82635eee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb0540bcdcf840fc827256983a122092" is_pre_election: true
I20260620 19:14:01.425629 29439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "587330cb1729499b986887e0149fdd13" candidate_uuid: "337d18064754461e9de8897a82635eee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97260049ae6b4537892ea05bbcc6af7d" is_pre_election: true
I20260620 19:14:01.425906 29439 raft_consensus.cc:2454] T 587330cb1729499b986887e0149fdd13 P 97260049ae6b4537892ea05bbcc6af7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 337d18064754461e9de8897a82635eee in term 0.
I20260620 19:14:01.425913 29171 raft_consensus.cc:2454] T 587330cb1729499b986887e0149fdd13 P eb0540bcdcf840fc827256983a122092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 337d18064754461e9de8897a82635eee in term 0.
I20260620 19:14:01.426362 29240 leader_election.cc:302] T 587330cb1729499b986887e0149fdd13 P 337d18064754461e9de8897a82635eee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 337d18064754461e9de8897a82635eee, 97260049ae6b4537892ea05bbcc6af7d; no voters:
I20260620 19:14:01.426687 29510 raft_consensus.cc:2781] T 587330cb1729499b986887e0149fdd13 P 337d18064754461e9de8897a82635eee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:14:01.426780 29510 raft_consensus.cc:482] T 587330cb1729499b986887e0149fdd13 P 337d18064754461e9de8897a82635eee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:14:01.426831 29510 raft_consensus.cc:3037] T 587330cb1729499b986887e0149fdd13 P 337d18064754461e9de8897a82635eee [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:14:01.427909 29510 raft_consensus.cc:504] T 587330cb1729499b986887e0149fdd13 P 337d18064754461e9de8897a82635eee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "337d18064754461e9de8897a82635eee" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 46401 } } peers { permanent_uuid: "eb0540bcdcf840fc827256983a122092" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 37971 } } peers { permanent_uuid: "97260049ae6b4537892ea05bbcc6af7d" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34359 } }
I20260620 19:14:01.428259 29510 leader_election.cc:288] T 587330cb1729499b986887e0149fdd13 P 337d18064754461e9de8897a82635eee [CANDIDATE]: Term 1 election: Requested vote from peers eb0540bcdcf840fc827256983a122092 (127.9.155.65:37971), 97260049ae6b4537892ea05bbcc6af7d (127.9.155.67:34359)
I20260620 19:14:01.428628 29171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "587330cb1729499b986887e0149fdd13" candidate_uuid: "337d18064754461e9de8897a82635eee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb0540bcdcf840fc827256983a122092"
I20260620 19:14:01.428628 29439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "587330cb1729499b986887e0149fdd13" candidate_uuid: "337d18064754461e9de8897a82635eee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97260049ae6b4537892ea05bbcc6af7d"
I20260620 19:14:01.428767 29171 raft_consensus.cc:3037] T 587330cb1729499b986887e0149fdd13 P eb0540bcdcf840fc827256983a122092 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:14:01.428757 29439 raft_consensus.cc:3037] T 587330cb1729499b986887e0149fdd13 P 97260049ae6b4537892ea05bbcc6af7d [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:14:01.429991 29171 raft_consensus.cc:2454] T 587330cb1729499b986887e0149fdd13 P eb0540bcdcf840fc827256983a122092 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 337d18064754461e9de8897a82635eee in term 1.
I20260620 19:14:01.429992 29439 raft_consensus.cc:2454] T 587330cb1729499b986887e0149fdd13 P 97260049ae6b4537892ea05bbcc6af7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 337d18064754461e9de8897a82635eee in term 1.
I20260620 19:14:01.430322 29240 leader_election.cc:302] T 587330cb1729499b986887e0149fdd13 P 337d18064754461e9de8897a82635eee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 337d18064754461e9de8897a82635eee, 97260049ae6b4537892ea05bbcc6af7d; no voters:
I20260620 19:14:01.430485 29510 raft_consensus.cc:2781] T 587330cb1729499b986887e0149fdd13 P 337d18064754461e9de8897a82635eee [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:14:01.430692 29510 raft_consensus.cc:686] T 587330cb1729499b986887e0149fdd13 P 337d18064754461e9de8897a82635eee [term 1 LEADER]: Becoming Leader. State: Replica: 337d18064754461e9de8897a82635eee, State: Running, Role: LEADER
I20260620 19:14:01.431018 29510 consensus_queue.cc:237] T 587330cb1729499b986887e0149fdd13 P 337d18064754461e9de8897a82635eee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "337d18064754461e9de8897a82635eee" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 46401 } } peers { permanent_uuid: "eb0540bcdcf840fc827256983a122092" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 37971 } } peers { permanent_uuid: "97260049ae6b4537892ea05bbcc6af7d" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34359 } }
I20260620 19:14:01.434114 29025 catalog_manager.cc:5495] T 587330cb1729499b986887e0149fdd13 P 337d18064754461e9de8897a82635eee reported cstate change: term changed from 0 to 1, leader changed from <none> to 337d18064754461e9de8897a82635eee (127.9.155.66). New cstate: current_term: 1 leader_uuid: "337d18064754461e9de8897a82635eee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "337d18064754461e9de8897a82635eee" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 46401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb0540bcdcf840fc827256983a122092" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 37971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97260049ae6b4537892ea05bbcc6af7d" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34359 } health_report { overall_health: UNKNOWN } } }
I20260620 19:14:01.444566 29497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:01.441218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40534 (local address 127.9.155.66:46401)
0620 19:14:01.441337 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:01.441341 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:01.441354 (+ 13us) server_negotiation.cc:407] Connection header received
0620 19:14:01.441488 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:01.441490 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:01.441543 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:01.441625 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:14:01.441861 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:01.442343 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:01.443199 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:01.443414 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:01.443494 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:01.443731 (+ 237us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:14:01.444138 (+ 407us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:14:01.444260 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:01.444362 (+ 102us) server_negotiation.cc:299] Negotiation successful
0620 19:14:01.444413 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 19:14:01.450220 29439 raft_consensus.cc:1261] T 587330cb1729499b986887e0149fdd13 P 97260049ae6b4537892ea05bbcc6af7d [term 1 FOLLOWER]: Refusing update from remote peer 337d18064754461e9de8897a82635eee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:14:01.450227 29171 raft_consensus.cc:1261] T 587330cb1729499b986887e0149fdd13 P eb0540bcdcf840fc827256983a122092 [term 1 FOLLOWER]: Refusing update from remote peer 337d18064754461e9de8897a82635eee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:14:01.450877 29510 consensus_queue.cc:1035] T 587330cb1729499b986887e0149fdd13 P 337d18064754461e9de8897a82635eee [LEADER]: Connected to new peer: Peer: permanent_uuid: "97260049ae6b4537892ea05bbcc6af7d" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 34359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:14:01.451058 29512 consensus_queue.cc:1035] T 587330cb1729499b986887e0149fdd13 P 337d18064754461e9de8897a82635eee [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb0540bcdcf840fc827256983a122092" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 37971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:14:01.458494 29516 mvcc.cc:205] Tried to move back new op lower bound from 7299001718572949504 to 7299001718503882752. Current Snapshot: MvccSnapshot[applied={T|T < 7299001718572949504}]
I20260620 19:14:01.459975 29515 mvcc.cc:205] Tried to move back new op lower bound from 7299001718572949504 to 7299001718503882752. Current Snapshot: MvccSnapshot[applied={T|T < 7299001718572949504}]
I20260620 19:14:01.467027 29025 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57326:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:14:01.467201 29025 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57326:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:14:01.469789 29025 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 31548f16d78f4af1a70ec7a47dad2e9a: Sending DeleteTablet for 3 replicas of tablet 587330cb1729499b986887e0149fdd13
I20260620 19:14:01.470570 29419 tablet_service.cc:1514] Processing DeleteTablet for tablet 587330cb1729499b986887e0149fdd13 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:14:01 UTC) from {username='kudu', principal='kudu/127.9.155.126@KRBTEST.COM'} at 127.0.0.1:45286
I20260620 19:14:01.470567 29151 tablet_service.cc:1514] Processing DeleteTablet for tablet 587330cb1729499b986887e0149fdd13 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:14:01 UTC) from {username='kudu', principal='kudu/127.9.155.126@KRBTEST.COM'} at 127.0.0.1:35346
I20260620 19:14:01.471072 29285 tablet_service.cc:1514] Processing DeleteTablet for tablet 587330cb1729499b986887e0149fdd13 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:14:01 UTC) from {username='kudu', principal='kudu/127.9.155.126@KRBTEST.COM'} at 127.0.0.1:40526
I20260620 19:14:01.471104 29525 tablet_replica.cc:330] T 587330cb1729499b986887e0149fdd13 P 97260049ae6b4537892ea05bbcc6af7d: stopping tablet replica
I20260620 19:14:01.471442 29525 raft_consensus.cc:2229] T 587330cb1729499b986887e0149fdd13 P 97260049ae6b4537892ea05bbcc6af7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:14:01.471499 29527 tablet_replica.cc:330] T 587330cb1729499b986887e0149fdd13 P 337d18064754461e9de8897a82635eee: stopping tablet replica
I20260620 19:14:01.471786 29525 raft_consensus.cc:2258] T 587330cb1729499b986887e0149fdd13 P 97260049ae6b4537892ea05bbcc6af7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:14:01.471956 29527 raft_consensus.cc:2229] T 587330cb1729499b986887e0149fdd13 P 337d18064754461e9de8897a82635eee [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:14:01.472182 29526 tablet_replica.cc:330] T 587330cb1729499b986887e0149fdd13 P eb0540bcdcf840fc827256983a122092: stopping tablet replica
I20260620 19:14:01.472265 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 29088
I20260620 19:14:01.473042 29527 raft_consensus.cc:2258] T 587330cb1729499b986887e0149fdd13 P 337d18064754461e9de8897a82635eee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:14:01.473852 29525 ts_tablet_manager.cc:1905] T 587330cb1729499b986887e0149fdd13 P 97260049ae6b4537892ea05bbcc6af7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:14:01.474362 29527 ts_tablet_manager.cc:1905] T 587330cb1729499b986887e0149fdd13 P 337d18064754461e9de8897a82635eee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:14:01.476604 29525 ts_tablet_manager.cc:1918] T 587330cb1729499b986887e0149fdd13 P 97260049ae6b4537892ea05bbcc6af7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:14:01.476711 29525 log.cc:1196] T 587330cb1729499b986887e0149fdd13 P 97260049ae6b4537892ea05bbcc6af7d: Deleting WAL directory at /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/minicluster-data/ts-2/wal/wals/587330cb1729499b986887e0149fdd13
I20260620 19:14:01.477016 29525 ts_tablet_manager.cc:1939] T 587330cb1729499b986887e0149fdd13 P 97260049ae6b4537892ea05bbcc6af7d: Deleting consensus metadata
I20260620 19:14:01.477121 29527 ts_tablet_manager.cc:1918] T 587330cb1729499b986887e0149fdd13 P 337d18064754461e9de8897a82635eee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:14:01.477258 29527 log.cc:1196] T 587330cb1729499b986887e0149fdd13 P 337d18064754461e9de8897a82635eee: Deleting WAL directory at /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982595231230-9837-0/minicluster-data/ts-1/wal/wals/587330cb1729499b986887e0149fdd13
I20260620 19:14:01.477591 29527 ts_tablet_manager.cc:1939] T 587330cb1729499b986887e0149fdd13 P 337d18064754461e9de8897a82635eee: Deleting consensus metadata
I20260620 19:14:01.478272 29012 catalog_manager.cc:4841] TS 97260049ae6b4537892ea05bbcc6af7d (127.9.155.67:34359): tablet 587330cb1729499b986887e0149fdd13 (table test-table [id=d8a82dbdd183443bba79670450ebcd9e]) successfully deleted
I20260620 19:14:01.478722 29012 catalog_manager.cc:4841] TS 337d18064754461e9de8897a82635eee (127.9.155.66:46401): tablet 587330cb1729499b986887e0149fdd13 (table test-table [id=d8a82dbdd183443bba79670450ebcd9e]) successfully deleted
W20260620 19:14:01.479125 29011 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.155.65:37971 (error 108)
W20260620 19:14:01.480492 29496 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:14:01.479853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.65:37971 (local address 127.0.0.1:35358)
0620 19:14:01.479972 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:01.479987 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 19:14:01.480059 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:14:01.480396 (+ 337us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.65:37971: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 19:14:01.480625 29011 catalog_manager.cc:4584] TS eb0540bcdcf840fc827256983a122092 (127.9.155.65:37971): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 587330cb1729499b986887e0149fdd13: Network error: Client connection negotiation failed: client connection to 127.9.155.65:37971: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 19:14:01.480919 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 29222
I20260620 19:14:01.485975 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 29356
I20260620 19:14:01.491081 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 28994
2026-06-20T19:14:01Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2177 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-20T19:14:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:14:01Z Disabled control of system clock
I20260620 19:14:01.519479 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982595231230-9837-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982595231230-9837-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982595231230-9837-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:35313
--webserver_interface=127.9.155.126
--webserver_port=0
--builtin_ntp_servers=127.9.155.84:37015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.155.126:35313
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 19:14:01.629458 29532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:14:01.629712 29532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:14:01.629747 29532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:14:01.633657 29532 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:14:01.633713 29532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:14:01.633726 29532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:14:01.633742 29532 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:14:01.633755 29532 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:14:01.638554 29532 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:37015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982595231230-9837-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982595231230-9837-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.155.126:35313
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:35313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.155.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29532
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982595231230-9837-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:14:01.639554 29532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:14:01.640858 29532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:14:01.649026 29538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:14:01.649235 29532 server_base.cc:1034] running on GCE node
W20260620 19:14:01.649132 29537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:14:01.649030 29540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:14:01.649904 29532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:14:01.651003 29532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:14:01.652194 29532 hybrid_clock.cc:648] HybridClock initialized: now 1781982841652171 us; error 36 us; skew 500 ppm
I20260620 19:14:01.654070 29532 webserver.cc:460] Webserver started at http://127.9.155.126:37133/ using document root <none> and password file <none>
I20260620 19:14:01.654623 29532 fs_manager.cc:359] Metadata directory not provided
I20260620 19:14:01.654691 29532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:14:01.654882 29532 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:14:01.656929 29532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982595231230-9837-0/minicluster-data/master-0/data/instance:
uuid: "ff192e1725064a14854993710ece4abd"
format_stamp: "Formatted at 2026-06-20 19:14:01 on dist-test-slave-n3fx"
server_key: "14f4df2e7d8c6d3a0a80e2f062342471"
server_key_iv: "de51174f50293f072c4074e584e5fc33"
server_key_version: "encryptionkey@0"
I20260620 19:14:01.657449 29532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982595231230-9837-0/minicluster-data/master-0/wal/instance:
uuid: "ff192e1725064a14854993710ece4abd"
format_stamp: "Formatted at 2026-06-20 19:14:01 on dist-test-slave-n3fx"
server_key: "14f4df2e7d8c6d3a0a80e2f062342471"
server_key_iv: "de51174f50293f072c4074e584e5fc33"
server_key_version: "encryptionkey@0"
I20260620 19:14:01.661036 29532 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 19:14:01.663411 29546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:01.664746 29532 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 19:14:01.664881 29532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982595231230-9837-0/minicluster-data/master-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982595231230-9837-0/minicluster-data/master-0/wal
uuid: "ff192e1725064a14854993710ece4abd"
format_stamp: "Formatted at 2026-06-20 19:14:01 on dist-test-slave-n3fx"
server_key: "14f4df2e7d8c6d3a0a80e2f062342471"
server_key_iv: "de51174f50293f072c4074e584e5fc33"
server_key_version: "encryptionkey@0"
I20260620 19:14:01.664979 29532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982595231230-9837-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982595231230-9837-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982595231230-9837-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:14:01.675145 29532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:14:01.675822 29532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:14:01.675989 29532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:14:01.682448 29532 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.126:35313
I20260620 19:14:01.682467 29598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.126:35313 every 8 connection(s)
I20260620 19:14:01.683333 29532 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
I20260620 19:14:01.685209 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 29532
I20260620 19:14:01.685338 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982595231230-9837-0/minicluster-data/master-0/wal/instance
I20260620 19:14:01.685643 9837 external_mini_cluster.cc:1428] Setting key 3edef50457a6471020aac8da481e0e5b
I20260620 19:14:01.687597 29599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:14:01.693208 29599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff192e1725064a14854993710ece4abd: Bootstrap starting.
I20260620 19:14:01.694857 29602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:01.686966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52186 (local address 127.9.155.126:35313)
0620 19:14:01.687616 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:01.687631 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:01.687672 (+ 41us) server_negotiation.cc:407] Connection header received
0620 19:14:01.688403 (+ 731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:01.688423 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:01.688839 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:01.689162 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:01.689572 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:01.690614 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:01.691498 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:01.691959 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:01.692123 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:01.693109 (+ 986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:01.693114 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:01.693127 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:14:01.693549 (+ 422us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:01.693612 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:14:01.693633 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:01.693717 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:01.693855 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 19:14:01.694125 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":141,"threads_started":1}
I20260620 19:14:01.695586 29599 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff192e1725064a14854993710ece4abd: Neither blocks nor log segments found. Creating new log.
I20260620 19:14:01.696395 29599 log.cc:824] T 00000000000000000000000000000000 P ff192e1725064a14854993710ece4abd: Log is configured to *not* fsync() on all Append() calls
I20260620 19:14:01.698249 29599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff192e1725064a14854993710ece4abd: No bootstrap required, opened a new log
I20260620 19:14:01.702023 29599 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff192e1725064a14854993710ece4abd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff192e1725064a14854993710ece4abd" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 35313 } }
I20260620 19:14:01.702322 29599 raft_consensus.cc:374] T 00000000000000000000000000000000 P ff192e1725064a14854993710ece4abd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:14:01.702387 29599 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff192e1725064a14854993710ece4abd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff192e1725064a14854993710ece4abd, State: Initialized, Role: FOLLOWER
I20260620 19:14:01.702823 29599 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff192e1725064a14854993710ece4abd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff192e1725064a14854993710ece4abd" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 35313 } }
I20260620 19:14:01.702952 29599 raft_consensus.cc:388] T 00000000000000000000000000000000 P ff192e1725064a14854993710ece4abd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:14:01.703017 29599 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff192e1725064a14854993710ece4abd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:14:01.703115 29599 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff192e1725064a14854993710ece4abd [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:14:01.704079 29599 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff192e1725064a14854993710ece4abd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff192e1725064a14854993710ece4abd" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 35313 } }
I20260620 19:14:01.704427 29599 leader_election.cc:302] T 00000000000000000000000000000000 P ff192e1725064a14854993710ece4abd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff192e1725064a14854993710ece4abd; no voters:
I20260620 19:14:01.704713 29599 leader_election.cc:288] T 00000000000000000000000000000000 P ff192e1725064a14854993710ece4abd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:14:01.704842 29604 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff192e1725064a14854993710ece4abd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:14:01.705049 29604 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff192e1725064a14854993710ece4abd [term 1 LEADER]: Becoming Leader. State: Replica: ff192e1725064a14854993710ece4abd, State: Running, Role: LEADER
I20260620 19:14:01.705339 29604 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff192e1725064a14854993710ece4abd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff192e1725064a14854993710ece4abd" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 35313 } }
I20260620 19:14:01.705857 29599 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff192e1725064a14854993710ece4abd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:14:01.706864 29606 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff192e1725064a14854993710ece4abd [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff192e1725064a14854993710ece4abd. Latest consensus state: current_term: 1 leader_uuid: "ff192e1725064a14854993710ece4abd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff192e1725064a14854993710ece4abd" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 35313 } } }
I20260620 19:14:01.706987 29606 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff192e1725064a14854993710ece4abd [sys.catalog]: This master's current role is: LEADER
I20260620 19:14:01.707288 29605 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff192e1725064a14854993710ece4abd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff192e1725064a14854993710ece4abd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff192e1725064a14854993710ece4abd" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 35313 } } }
I20260620 19:14:01.707383 29605 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff192e1725064a14854993710ece4abd [sys.catalog]: This master's current role is: LEADER
I20260620 19:14:01.707659 29613 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:14:01.710716 29613 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:14:01.716001 29613 catalog_manager.cc:1347] Generated new cluster ID: a1b3572ac3dd45f7857265f5cfe61a9a
I20260620 19:14:01.716073 29613 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:14:01.736698 29613 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:14:01.737672 29613 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:14:01.741302 29613 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ff192e1725064a14854993710ece4abd: Generated new TSK 0
I20260620 19:14:01.741950 29613 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:14:01.751538 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982595231230-9837-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982595231230-9837-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982595231230-9837-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.65:0
--local_ip_for_outbound_sockets=127.9.155.65
--webserver_interface=127.9.155.65
--webserver_port=0
--tserver_master_addrs=127.9.155.126:35313
--builtin_ntp_servers=127.9.155.84:37015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 19:14:01.864328 29623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:14:01.864594 29623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:14:01.864632 29623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:14:01.868883 29623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:14:01.868947 29623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:14:01.869010 29623 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.65
I20260620 19:14:01.873898 29623 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:37015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982595231230-9837-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982595231230-9837-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.155.65
--webserver_port=0
--tserver_master_addrs=127.9.155.126:35313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29623
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.65
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982595231230-9837-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:14:01.874976 29623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:14:01.876436 29623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:14:01.886415 29628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:14:01.886415 29631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:14:01.886458 29623 server_base.cc:1034] running on GCE node
W20260620 19:14:01.886411 29629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:14:01.887300 29623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:14:01.888005 29623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:14:01.889197 29623 hybrid_clock.cc:648] HybridClock initialized: now 1781982841889173 us; error 46 us; skew 500 ppm
I20260620 19:14:01.891206 29623 webserver.cc:460] Webserver started at http://127.9.155.65:40531/ using document root <none> and password file <none>
I20260620 19:14:01.891772 29623 fs_manager.cc:359] Metadata directory not provided
I20260620 19:14:01.891834 29623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:14:01.892028 29623 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:14:01.894126 29623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982595231230-9837-0/minicluster-data/ts-0/data/instance:
uuid: "d317ee71d7a54e67b3da41f11316744d"
format_stamp: "Formatted at 2026-06-20 19:14:01 on dist-test-slave-n3fx"
server_key: "681d1f8104c7f0e27c64d82cd29a4eae"
server_key_iv: "eac645eb58089ed756e238029ec332a4"
server_key_version: "encryptionkey@0"
I20260620 19:14:01.894652 29623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982595231230-9837-0/minicluster-data/ts-0/wal/instance:
uuid: "d317ee71d7a54e67b3da41f11316744d"
format_stamp: "Formatted at 2026-06-20 19:14:01 on dist-test-slave-n3fx"
server_key: "681d1f8104c7f0e27c64d82cd29a4eae"
server_key_iv: "eac645eb58089ed756e238029ec332a4"
server_key_version: "encryptionkey@0"
I20260620 19:14:01.898185 29623 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 19:14:01.900578 29637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:01.901719 29623 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:14:01.901827 29623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982595231230-9837-0/minicluster-data/ts-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982595231230-9837-0/minicluster-data/ts-0/wal
uuid: "d317ee71d7a54e67b3da41f11316744d"
format_stamp: "Formatted at 2026-06-20 19:14:01 on dist-test-slave-n3fx"
server_key: "681d1f8104c7f0e27c64d82cd29a4eae"
server_key_iv: "eac645eb58089ed756e238029ec332a4"
server_key_version: "encryptionkey@0"
I20260620 19:14:01.901923 29623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982595231230-9837-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982595231230-9837-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982595231230-9837-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:14:01.917119 29623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:14:01.917870 29623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:14:01.918059 29623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:14:01.918651 29623 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:14:01.919576 29623 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:14:01.919621 29623 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:01.919660 29623 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:14:01.919695 29623 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:01.928371 29623 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.65:42941
I20260620 19:14:01.928398 29750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.65:42941 every 8 connection(s)
I20260620 19:14:01.929230 29623 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982595231230-9837-0/minicluster-data/ts-0/data/info.pb
I20260620 19:14:01.937708 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 29623
I20260620 19:14:01.937806 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982595231230-9837-0/minicluster-data/ts-0/wal/instance
I20260620 19:14:01.938107 9837 external_mini_cluster.cc:1428] Setting key 423735ab2eeddac8564ef206f8b06484
I20260620 19:14:01.938109 29602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:01.930918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.65:50527 (local address 127.9.155.126:35313)
0620 19:14:01.931057 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:01.931061 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:01.932107 (+ 1046us) server_negotiation.cc:407] Connection header received
0620 19:14:01.933512 (+ 1405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:01.933516 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:01.933578 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:01.933651 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:01.934520 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:01.935052 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:01.936244 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:01.936463 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:01.936554 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:01.937580 (+ 1026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:01.937583 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:01.937586 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:14:01.937599 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:01.937617 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:14:01.937625 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:01.937682 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:01.937870 (+ 188us) server_negotiation.cc:299] Negotiation successful
0620 19:14:01.937962 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 19:14:01.938953 29753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:01.931307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:35313 (local address 127.9.155.65:50527)
0620 19:14:01.931943 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:01.931983 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 19:14:01.933302 (+ 1319us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:14:01.933832 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:14:01.933839 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:14:01.933921 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:14:01.934383 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:01.934395 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:01.935172 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:01.935176 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:01.936090 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:01.936100 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:01.936603 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:01.936607 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:01.936769 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:01.937335 (+ 566us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:14:01.937337 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:14:01.937374 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:14:01.937435 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:14:01.937697 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:14:01.937706 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:14:01.937712 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 19:14:01.937826 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 19:14:01.938073 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":109,"threads_started":1}
I20260620 19:14:01.940026 29751 heartbeater.cc:347] Connected to a master server at 127.9.155.126:35313
I20260620 19:14:01.940062 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982595231230-9837-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982595231230-9837-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982595231230-9837-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.66:0
--local_ip_for_outbound_sockets=127.9.155.66
--webserver_interface=127.9.155.66
--webserver_port=0
--tserver_master_addrs=127.9.155.126:35313
--builtin_ntp_servers=127.9.155.84:37015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 19:14:01.940299 29751 heartbeater.cc:464] Registering TS with master...
I20260620 19:14:01.940783 29751 heartbeater.cc:511] Master 127.9.155.126:35313 requested a full tablet report, sending...
I20260620 19:14:01.942276 29563 ts_manager.cc:194] Registered new tserver with Master: d317ee71d7a54e67b3da41f11316744d (127.9.155.65:42941)
I20260620 19:14:01.943348 29563 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.155.65:50527
W20260620 19:14:02.052323 29754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:14:02.052603 29754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:14:02.052641 29754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:14:02.056677 29754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:14:02.056747 29754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:14:02.056805 29754 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.66
I20260620 19:14:02.061759 29754 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:37015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982595231230-9837-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982595231230-9837-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.155.66
--webserver_port=0
--tserver_master_addrs=127.9.155.126:35313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29754
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.66
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982595231230-9837-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:14:02.062845 29754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:14:02.064181 29754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:14:02.074326 29762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:14:02.074288 29760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:14:02.074337 29759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:14:02.074610 29754 server_base.cc:1034] running on GCE node
I20260620 19:14:02.075031 29754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:14:02.075695 29754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:14:02.076892 29754 hybrid_clock.cc:648] HybridClock initialized: now 1781982842076845 us; error 61 us; skew 500 ppm
I20260620 19:14:02.078855 29754 webserver.cc:460] Webserver started at http://127.9.155.66:42193/ using document root <none> and password file <none>
I20260620 19:14:02.079411 29754 fs_manager.cc:359] Metadata directory not provided
I20260620 19:14:02.079476 29754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:14:02.079656 29754 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:14:02.081754 29754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982595231230-9837-0/minicluster-data/ts-1/data/instance:
uuid: "68a4beea4ae947dcbf9453966598285d"
format_stamp: "Formatted at 2026-06-20 19:14:02 on dist-test-slave-n3fx"
server_key: "d9a2dc60c5697d547956a46ad99a91f2"
server_key_iv: "f5f1b14be0716ea015904732fc64a8e5"
server_key_version: "encryptionkey@0"
I20260620 19:14:02.082346 29754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982595231230-9837-0/minicluster-data/ts-1/wal/instance:
uuid: "68a4beea4ae947dcbf9453966598285d"
format_stamp: "Formatted at 2026-06-20 19:14:02 on dist-test-slave-n3fx"
server_key: "d9a2dc60c5697d547956a46ad99a91f2"
server_key_iv: "f5f1b14be0716ea015904732fc64a8e5"
server_key_version: "encryptionkey@0"
I20260620 19:14:02.086243 29754 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 19:14:02.088899 29768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:02.090183 29754 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:14:02.090301 29754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982595231230-9837-0/minicluster-data/ts-1/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982595231230-9837-0/minicluster-data/ts-1/wal
uuid: "68a4beea4ae947dcbf9453966598285d"
format_stamp: "Formatted at 2026-06-20 19:14:02 on dist-test-slave-n3fx"
server_key: "d9a2dc60c5697d547956a46ad99a91f2"
server_key_iv: "f5f1b14be0716ea015904732fc64a8e5"
server_key_version: "encryptionkey@0"
I20260620 19:14:02.090407 29754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982595231230-9837-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982595231230-9837-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982595231230-9837-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:14:02.100466 29754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:14:02.101142 29754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:14:02.101356 29754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:14:02.101954 29754 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:14:02.102903 29754 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:14:02.102954 29754 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:02.103009 29754 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:14:02.103037 29754 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:02.112403 29754 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.66:32913
I20260620 19:14:02.112433 29881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.66:32913 every 8 connection(s)
I20260620 19:14:02.113330 29754 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982595231230-9837-0/minicluster-data/ts-1/data/info.pb
I20260620 19:14:02.118170 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 29754
I20260620 19:14:02.118356 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982595231230-9837-0/minicluster-data/ts-1/wal/instance
I20260620 19:14:02.118640 9837 external_mini_cluster.cc:1428] Setting key f388f64aef43577e537c8e40f3b0bbd8
I20260620 19:14:02.120801 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982595231230-9837-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982595231230-9837-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982595231230-9837-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.155.67:0
--local_ip_for_outbound_sockets=127.9.155.67
--webserver_interface=127.9.155.67
--webserver_port=0
--tserver_master_addrs=127.9.155.126:35313
--builtin_ntp_servers=127.9.155.84:37015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 19:14:02.123428 29602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:02.115097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.66:51661 (local address 127.9.155.126:35313)
0620 19:14:02.115267 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:02.115272 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:02.116416 (+ 1144us) server_negotiation.cc:407] Connection header received
0620 19:14:02.117923 (+ 1507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:02.117927 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:02.117994 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:02.118101 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:02.119216 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:02.119822 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:02.121245 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:02.121565 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:02.121681 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:02.122832 (+ 1151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:02.122836 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:02.122840 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:14:02.122858 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:02.122886 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:14:02.122896 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:02.122978 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:02.123207 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 19:14:02.123276 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 19:14:02.124341 29884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:02.115560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:35313 (local address 127.9.155.66:51661)
0620 19:14:02.116257 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:02.116301 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 19:14:02.117637 (+ 1336us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:14:02.118330 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:14:02.118338 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:14:02.118461 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:14:02.119020 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:02.119050 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:02.119952 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:02.119955 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:02.121082 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:02.121094 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:02.121751 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:02.121754 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:02.121928 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:02.122540 (+ 612us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:14:02.122542 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:14:02.122585 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:14:02.122661 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:14:02.123004 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:14:02.123012 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:14:02.123017 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 19:14:02.123185 (+ 168us) client_negotiation.cc:240] Negotiation successful
0620 19:14:02.123438 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":105,"threads_started":1}
I20260620 19:14:02.125541 29882 heartbeater.cc:347] Connected to a master server at 127.9.155.126:35313
I20260620 19:14:02.125767 29882 heartbeater.cc:464] Registering TS with master...
I20260620 19:14:02.126266 29882 heartbeater.cc:511] Master 127.9.155.126:35313 requested a full tablet report, sending...
I20260620 19:14:02.127308 29563 ts_manager.cc:194] Registered new tserver with Master: 68a4beea4ae947dcbf9453966598285d (127.9.155.66:32913)
I20260620 19:14:02.127869 29563 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.155.66:51661
W20260620 19:14:02.231259 29885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:14:02.231518 29885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:14:02.231554 29885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:14:02.235546 29885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:14:02.235622 29885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:14:02.235679 29885 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.155.67
I20260620 19:14:02.240571 29885 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:37015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982595231230-9837-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982595231230-9837-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.155.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.155.67
--webserver_port=0
--tserver_master_addrs=127.9.155.126:35313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29885
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.155.67
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982595231230-9837-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:14:02.241755 29885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:14:02.243155 29885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:14:02.252494 29891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:14:02.252672 29885 server_base.cc:1034] running on GCE node
W20260620 19:14:02.252494 29890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:14:02.252530 29893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:14:02.253091 29885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:14:02.253731 29885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:14:02.254887 29885 hybrid_clock.cc:648] HybridClock initialized: now 1781982842254870 us; error 31 us; skew 500 ppm
I20260620 19:14:02.256683 29885 webserver.cc:460] Webserver started at http://127.9.155.67:33433/ using document root <none> and password file <none>
I20260620 19:14:02.257253 29885 fs_manager.cc:359] Metadata directory not provided
I20260620 19:14:02.257323 29885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:14:02.257505 29885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:14:02.259537 29885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982595231230-9837-0/minicluster-data/ts-2/data/instance:
uuid: "e7ea684fba0a468c99a18c3edd39ca7c"
format_stamp: "Formatted at 2026-06-20 19:14:02 on dist-test-slave-n3fx"
server_key: "8b10fc19fc0e7481c73aa4c2b0f9b9d1"
server_key_iv: "f9671e35b08de7cb94953b279fcaf7ce"
server_key_version: "encryptionkey@0"
I20260620 19:14:02.260083 29885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982595231230-9837-0/minicluster-data/ts-2/wal/instance:
uuid: "e7ea684fba0a468c99a18c3edd39ca7c"
format_stamp: "Formatted at 2026-06-20 19:14:02 on dist-test-slave-n3fx"
server_key: "8b10fc19fc0e7481c73aa4c2b0f9b9d1"
server_key_iv: "f9671e35b08de7cb94953b279fcaf7ce"
server_key_version: "encryptionkey@0"
I20260620 19:14:02.263727 29885 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 19:14:02.266273 29899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:02.267582 29885 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 19:14:02.267714 29885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982595231230-9837-0/minicluster-data/ts-2/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982595231230-9837-0/minicluster-data/ts-2/wal
uuid: "e7ea684fba0a468c99a18c3edd39ca7c"
format_stamp: "Formatted at 2026-06-20 19:14:02 on dist-test-slave-n3fx"
server_key: "8b10fc19fc0e7481c73aa4c2b0f9b9d1"
server_key_iv: "f9671e35b08de7cb94953b279fcaf7ce"
server_key_version: "encryptionkey@0"
I20260620 19:14:02.267827 29885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982595231230-9837-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982595231230-9837-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982595231230-9837-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:14:02.282459 29885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:14:02.283254 29885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:14:02.283428 29885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:14:02.284046 29885 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:14:02.285053 29885 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:14:02.285108 29885 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:02.285163 29885 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:14:02.285190 29885 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:02.293497 29885 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.67:32935
I20260620 19:14:02.293565 30012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.67:32935 every 8 connection(s)
I20260620 19:14:02.294449 29885 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982595231230-9837-0/minicluster-data/ts-2/data/info.pb
I20260620 19:14:02.297262 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 29885
I20260620 19:14:02.297369 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982595231230-9837-0/minicluster-data/ts-2/wal/instance
I20260620 19:14:02.297655 9837 external_mini_cluster.cc:1428] Setting key a13ad633d6245eabed108ee89ad393fb
I20260620 19:14:02.303815 29602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:02.296182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.67:44489 (local address 127.9.155.126:35313)
0620 19:14:02.296356 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:02.296361 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:02.297517 (+ 1156us) server_negotiation.cc:407] Connection header received
0620 19:14:02.299043 (+ 1526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:02.299045 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:02.299085 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:02.299150 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:02.300169 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:02.300723 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:02.301926 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:02.302143 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:02.302239 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:02.303319 (+ 1080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:02.303323 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:02.303325 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:14:02.303338 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:02.303356 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:14:02.303363 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:02.303430 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:02.303613 (+ 183us) server_negotiation.cc:299] Negotiation successful
0620 19:14:02.303666 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 19:14:02.304832 30015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:02.296635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:35313 (local address 127.9.155.67:44489)
0620 19:14:02.297359 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:02.297413 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 19:14:02.298856 (+ 1443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:14:02.299320 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:14:02.299329 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:14:02.299436 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:14:02.300002 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:02.300015 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:02.300843 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:02.300846 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:02.301800 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:02.301810 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:02.302327 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:02.302330 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:02.302478 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:02.303061 (+ 583us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:14:02.303063 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:14:02.303104 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:14:02.303172 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:14:02.303473 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:14:02.303481 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:14:02.303486 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 19:14:02.303606 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 19:14:02.303857 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":107,"threads_started":1}
I20260620 19:14:02.306025 30013 heartbeater.cc:347] Connected to a master server at 127.9.155.126:35313
I20260620 19:14:02.306294 30013 heartbeater.cc:464] Registering TS with master...
I20260620 19:14:02.306845 30013 heartbeater.cc:511] Master 127.9.155.126:35313 requested a full tablet report, sending...
I20260620 19:14:02.308048 29563 ts_manager.cc:194] Registered new tserver with Master: e7ea684fba0a468c99a18c3edd39ca7c (127.9.155.67:32935)
I20260620 19:14:02.308674 29563 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.155.67:44489
I20260620 19:14:02.311369 9837 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:14:02.316984 29602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:02.313285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52196 (local address 127.9.155.126:35313)
0620 19:14:02.313409 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:02.313414 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:02.313572 (+ 158us) server_negotiation.cc:407] Connection header received
0620 19:14:02.313685 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:02.313688 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:02.313755 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:02.313820 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:02.314094 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:02.314661 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:02.315502 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:02.315730 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:02.315816 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:02.316605 (+ 789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:02.316608 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:02.316611 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:14:02.316624 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:02.316644 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:14:02.316652 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:02.316718 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:02.316793 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 19:14:02.316846 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 19:14:02.320304 29563 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52196:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:14:02.324280 29563 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:14:02.337844 30026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:02.332261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36296 (local address 127.9.155.65:42941)
0620 19:14:02.332828 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:02.332839 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:02.332868 (+ 29us) server_negotiation.cc:407] Connection header received
0620 19:14:02.332945 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:02.332950 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:02.333161 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:02.333311 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:14:02.333752 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:02.334763 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:02.336453 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:02.337192 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:02.337294 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:02.337490 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:02.337592 (+ 102us) server_negotiation.cc:299] Negotiation successful
0620 19:14:02.337686 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":111,"threads_started":1}
I20260620 19:14:02.337841 30025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:02.332083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.65:42941 (local address 127.0.0.1:36296)
0620 19:14:02.332580 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:02.332627 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 19:14:02.332769 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:14:02.333487 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:14:02.333493 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:14:02.333517 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:14:02.333640 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:02.333645 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:02.334864 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:02.334867 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:02.336331 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:02.336342 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:02.337311 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:02.337315 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:02.337418 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:02.337453 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 19:14:02.337562 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 19:14:02.337682 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"mutex_wait_us":54,"spinlock_wait_cycles":8448,"thread_start_us":73,"threads_started":1}
I20260620 19:14:02.338419 30024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:02.332197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.66:32913 (local address 127.0.0.1:35304)
0620 19:14:02.332627 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:02.332642 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 19:14:02.332888 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:14:02.333918 (+ 1030us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:14:02.333921 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:14:02.333935 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:14:02.334044 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:02.334050 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:02.335263 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:02.335265 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:02.336490 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:02.336500 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:02.337750 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:02.337754 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:02.337848 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:02.337870 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 19:14:02.338214 (+ 344us) client_negotiation.cc:240] Negotiation successful
0620 19:14:02.338284 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"mutex_wait_us":23,"spinlock_wait_cycles":5632,"thread_start_us":111,"threads_started":1}
I20260620 19:14:02.338521 30028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:02.333133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35304 (local address 127.9.155.66:32913)
0620 19:14:02.333509 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:02.333520 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:02.333545 (+ 25us) server_negotiation.cc:407] Connection header received
0620 19:14:02.333635 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:02.333640 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:02.333804 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:02.333929 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:14:02.334177 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:02.335164 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:02.336613 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:02.337416 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:02.337517 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:02.337688 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:02.338263 (+ 575us) server_negotiation.cc:299] Negotiation successful
0620 19:14:02.338372 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":84,"threads_started":1}
I20260620 19:14:02.339215 30027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:02.332776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.67:32935 (local address 127.0.0.1:56206)
0620 19:14:02.333566 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:02.333579 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:14:02.333654 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:14:02.334536 (+ 882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:14:02.334539 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:14:02.334554 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:14:02.334639 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:02.334646 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:02.335974 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:02.335977 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:02.337281 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:02.337292 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:02.338686 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:02.338702 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:02.338934 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:02.338957 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 19:14:02.339027 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 19:14:02.339093 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":734,"mutex_wait_us":206,"thread_start_us":87,"threads_started":1}
I20260620 19:14:02.339609 30029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:02.333268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56206 (local address 127.9.155.67:32935)
0620 19:14:02.334191 (+ 923us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:02.334200 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:02.334217 (+ 17us) server_negotiation.cc:407] Connection header received
0620 19:14:02.334272 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:02.334275 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:02.334412 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:02.334541 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:14:02.334743 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:02.335850 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:02.337420 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:02.338147 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:02.338246 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:02.338406 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:02.339358 (+ 952us) server_negotiation.cc:299] Negotiation successful
0620 19:14:02.339457 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":828,"thread_start_us":78,"threads_started":1}
I20260620 19:14:02.341094 29685 tablet_service.cc:1467] Processing CreateTablet for tablet bdc83af3311f4ee5a7ceaa5c90fe9947 (DEFAULT_TABLE table=test-table [id=88546ffe51a849fab3634cc6bbfa49bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:14:02.341094 29816 tablet_service.cc:1467] Processing CreateTablet for tablet bdc83af3311f4ee5a7ceaa5c90fe9947 (DEFAULT_TABLE table=test-table [id=88546ffe51a849fab3634cc6bbfa49bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:14:02.341820 29947 tablet_service.cc:1467] Processing CreateTablet for tablet bdc83af3311f4ee5a7ceaa5c90fe9947 (DEFAULT_TABLE table=test-table [id=88546ffe51a849fab3634cc6bbfa49bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:14:02.342221 29816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdc83af3311f4ee5a7ceaa5c90fe9947. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:14:02.342221 29685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdc83af3311f4ee5a7ceaa5c90fe9947. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:14:02.342677 29947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdc83af3311f4ee5a7ceaa5c90fe9947. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:14:02.348683 30030 tablet_bootstrap.cc:492] T bdc83af3311f4ee5a7ceaa5c90fe9947 P e7ea684fba0a468c99a18c3edd39ca7c: Bootstrap starting.
I20260620 19:14:02.349968 30031 tablet_bootstrap.cc:492] T bdc83af3311f4ee5a7ceaa5c90fe9947 P 68a4beea4ae947dcbf9453966598285d: Bootstrap starting.
I20260620 19:14:02.351153 30030 tablet_bootstrap.cc:654] T bdc83af3311f4ee5a7ceaa5c90fe9947 P e7ea684fba0a468c99a18c3edd39ca7c: Neither blocks nor log segments found. Creating new log.
I20260620 19:14:02.351662 30031 tablet_bootstrap.cc:654] T bdc83af3311f4ee5a7ceaa5c90fe9947 P 68a4beea4ae947dcbf9453966598285d: Neither blocks nor log segments found. Creating new log.
I20260620 19:14:02.351856 30030 log.cc:824] T bdc83af3311f4ee5a7ceaa5c90fe9947 P e7ea684fba0a468c99a18c3edd39ca7c: Log is configured to *not* fsync() on all Append() calls
I20260620 19:14:02.352381 30031 log.cc:824] T bdc83af3311f4ee5a7ceaa5c90fe9947 P 68a4beea4ae947dcbf9453966598285d: Log is configured to *not* fsync() on all Append() calls
I20260620 19:14:02.352486 30032 tablet_bootstrap.cc:492] T bdc83af3311f4ee5a7ceaa5c90fe9947 P d317ee71d7a54e67b3da41f11316744d: Bootstrap starting.
I20260620 19:14:02.353977 30030 tablet_bootstrap.cc:492] T bdc83af3311f4ee5a7ceaa5c90fe9947 P e7ea684fba0a468c99a18c3edd39ca7c: No bootstrap required, opened a new log
I20260620 19:14:02.354251 30030 ts_tablet_manager.cc:1397] T bdc83af3311f4ee5a7ceaa5c90fe9947 P e7ea684fba0a468c99a18c3edd39ca7c: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260620 19:14:02.354887 30031 tablet_bootstrap.cc:492] T bdc83af3311f4ee5a7ceaa5c90fe9947 P 68a4beea4ae947dcbf9453966598285d: No bootstrap required, opened a new log
I20260620 19:14:02.354979 30032 tablet_bootstrap.cc:654] T bdc83af3311f4ee5a7ceaa5c90fe9947 P d317ee71d7a54e67b3da41f11316744d: Neither blocks nor log segments found. Creating new log.
I20260620 19:14:02.355069 30031 ts_tablet_manager.cc:1397] T bdc83af3311f4ee5a7ceaa5c90fe9947 P 68a4beea4ae947dcbf9453966598285d: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260620 19:14:02.355715 30032 log.cc:824] T bdc83af3311f4ee5a7ceaa5c90fe9947 P d317ee71d7a54e67b3da41f11316744d: Log is configured to *not* fsync() on all Append() calls
I20260620 19:14:02.357475 30032 tablet_bootstrap.cc:492] T bdc83af3311f4ee5a7ceaa5c90fe9947 P d317ee71d7a54e67b3da41f11316744d: No bootstrap required, opened a new log
I20260620 19:14:02.357662 30032 ts_tablet_manager.cc:1397] T bdc83af3311f4ee5a7ceaa5c90fe9947 P d317ee71d7a54e67b3da41f11316744d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 19:14:02.358639 30030 raft_consensus.cc:348] T bdc83af3311f4ee5a7ceaa5c90fe9947 P e7ea684fba0a468c99a18c3edd39ca7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d317ee71d7a54e67b3da41f11316744d" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 42941 } } peers { permanent_uuid: "68a4beea4ae947dcbf9453966598285d" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 32913 } } peers { permanent_uuid: "e7ea684fba0a468c99a18c3edd39ca7c" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 32935 } }
I20260620 19:14:02.358923 30030 raft_consensus.cc:374] T bdc83af3311f4ee5a7ceaa5c90fe9947 P e7ea684fba0a468c99a18c3edd39ca7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:14:02.359010 30030 raft_consensus.cc:729] T bdc83af3311f4ee5a7ceaa5c90fe9947 P e7ea684fba0a468c99a18c3edd39ca7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7ea684fba0a468c99a18c3edd39ca7c, State: Initialized, Role: FOLLOWER
I20260620 19:14:02.359277 30031 raft_consensus.cc:348] T bdc83af3311f4ee5a7ceaa5c90fe9947 P 68a4beea4ae947dcbf9453966598285d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d317ee71d7a54e67b3da41f11316744d" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 42941 } } peers { permanent_uuid: "68a4beea4ae947dcbf9453966598285d" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 32913 } } peers { permanent_uuid: "e7ea684fba0a468c99a18c3edd39ca7c" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 32935 } }
I20260620 19:14:02.359519 30031 raft_consensus.cc:374] T bdc83af3311f4ee5a7ceaa5c90fe9947 P 68a4beea4ae947dcbf9453966598285d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:14:02.359489 30030 consensus_queue.cc:260] T bdc83af3311f4ee5a7ceaa5c90fe9947 P e7ea684fba0a468c99a18c3edd39ca7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d317ee71d7a54e67b3da41f11316744d" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 42941 } } peers { permanent_uuid: "68a4beea4ae947dcbf9453966598285d" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 32913 } } peers { permanent_uuid: "e7ea684fba0a468c99a18c3edd39ca7c" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 32935 } }
I20260620 19:14:02.359604 30031 raft_consensus.cc:729] T bdc83af3311f4ee5a7ceaa5c90fe9947 P 68a4beea4ae947dcbf9453966598285d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a4beea4ae947dcbf9453966598285d, State: Initialized, Role: FOLLOWER
I20260620 19:14:02.360067 30031 consensus_queue.cc:260] T bdc83af3311f4ee5a7ceaa5c90fe9947 P 68a4beea4ae947dcbf9453966598285d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d317ee71d7a54e67b3da41f11316744d" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 42941 } } peers { permanent_uuid: "68a4beea4ae947dcbf9453966598285d" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 32913 } } peers { permanent_uuid: "e7ea684fba0a468c99a18c3edd39ca7c" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 32935 } }
I20260620 19:14:02.360570 30030 ts_tablet_manager.cc:1428] T bdc83af3311f4ee5a7ceaa5c90fe9947 P e7ea684fba0a468c99a18c3edd39ca7c: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260620 19:14:02.360986 30013 heartbeater.cc:503] Master 127.9.155.126:35313 was elected leader, sending a full tablet report...
I20260620 19:14:02.361337 30031 ts_tablet_manager.cc:1428] T bdc83af3311f4ee5a7ceaa5c90fe9947 P 68a4beea4ae947dcbf9453966598285d: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260620 19:14:02.361631 29882 heartbeater.cc:503] Master 127.9.155.126:35313 was elected leader, sending a full tablet report...
I20260620 19:14:02.361624 30032 raft_consensus.cc:348] T bdc83af3311f4ee5a7ceaa5c90fe9947 P d317ee71d7a54e67b3da41f11316744d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d317ee71d7a54e67b3da41f11316744d" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 42941 } } peers { permanent_uuid: "68a4beea4ae947dcbf9453966598285d" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 32913 } } peers { permanent_uuid: "e7ea684fba0a468c99a18c3edd39ca7c" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 32935 } }
I20260620 19:14:02.361865 30032 raft_consensus.cc:374] T bdc83af3311f4ee5a7ceaa5c90fe9947 P d317ee71d7a54e67b3da41f11316744d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:14:02.361913 30032 raft_consensus.cc:729] T bdc83af3311f4ee5a7ceaa5c90fe9947 P d317ee71d7a54e67b3da41f11316744d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d317ee71d7a54e67b3da41f11316744d, State: Initialized, Role: FOLLOWER
I20260620 19:14:02.362298 30032 consensus_queue.cc:260] T bdc83af3311f4ee5a7ceaa5c90fe9947 P d317ee71d7a54e67b3da41f11316744d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d317ee71d7a54e67b3da41f11316744d" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 42941 } } peers { permanent_uuid: "68a4beea4ae947dcbf9453966598285d" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 32913 } } peers { permanent_uuid: "e7ea684fba0a468c99a18c3edd39ca7c" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 32935 } }
I20260620 19:14:02.363214 30032 ts_tablet_manager.cc:1428] T bdc83af3311f4ee5a7ceaa5c90fe9947 P d317ee71d7a54e67b3da41f11316744d: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 19:14:02.363396 29751 heartbeater.cc:503] Master 127.9.155.126:35313 was elected leader, sending a full tablet report...
W20260620 19:14:02.364171 29883 tablet.cc:2369] T bdc83af3311f4ee5a7ceaa5c90fe9947 P 68a4beea4ae947dcbf9453966598285d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:14:02.384536 30038 raft_consensus.cc:482] T bdc83af3311f4ee5a7ceaa5c90fe9947 P d317ee71d7a54e67b3da41f11316744d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:14:02.384737 30038 raft_consensus.cc:504] T bdc83af3311f4ee5a7ceaa5c90fe9947 P d317ee71d7a54e67b3da41f11316744d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d317ee71d7a54e67b3da41f11316744d" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 42941 } } peers { permanent_uuid: "68a4beea4ae947dcbf9453966598285d" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 32913 } } peers { permanent_uuid: "e7ea684fba0a468c99a18c3edd39ca7c" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 32935 } }
I20260620 19:14:02.385856 30038 leader_election.cc:288] T bdc83af3311f4ee5a7ceaa5c90fe9947 P d317ee71d7a54e67b3da41f11316744d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68a4beea4ae947dcbf9453966598285d (127.9.155.66:32913), e7ea684fba0a468c99a18c3edd39ca7c (127.9.155.67:32935)
I20260620 19:14:02.390218 29753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:02.386085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.66:32913 (local address 127.9.155.65:55253)
0620 19:14:02.386218 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:02.386231 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:14:02.386319 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:14:02.386618 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:14:02.386622 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:14:02.386645 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:14:02.386777 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:02.386784 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:02.387815 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:02.387818 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:02.389011 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:02.389022 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:02.389843 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:02.389845 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:02.389922 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:02.389943 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 19:14:02.390000 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 19:14:02.390074 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 19:14:02.390290 30028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:02.386253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.65:55253 (local address 127.9.155.66:32913)
0620 19:14:02.386384 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:02.386389 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:02.386402 (+ 13us) server_negotiation.cc:407] Connection header received
0620 19:14:02.386455 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:02.386458 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:02.386506 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:02.386582 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:14:02.386905 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:02.387710 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:02.389159 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:02.389724 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:02.389850 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:02.389901 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:02.390103 (+ 202us) server_negotiation.cc:299] Negotiation successful
0620 19:14:02.390178 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 19:14:02.390461 30039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:02.386423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.67:32935 (local address 127.9.155.65:49015)
0620 19:14:02.386815 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:02.386828 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:14:02.386920 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:14:02.387154 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:14:02.387156 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:14:02.387168 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:14:02.387290 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:02.387294 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:02.388112 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:02.388116 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:02.389247 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:02.389255 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:02.390069 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:02.390072 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:02.390157 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:02.390176 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 19:14:02.390277 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 19:14:02.390339 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":114,"threads_started":1}
I20260620 19:14:02.390477 30029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:02.386541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.155.65:49015 (local address 127.9.155.67:32935)
0620 19:14:02.386673 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:02.386676 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:02.386895 (+ 219us) server_negotiation.cc:407] Connection header received
0620 19:14:02.387001 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:02.387004 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:02.387063 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:02.387140 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:14:02.387379 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:02.388013 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:02.389364 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:02.389952 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:02.390061 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:02.390104 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:02.390298 (+ 194us) server_negotiation.cc:299] Negotiation successful
0620 19:14:02.390368 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 19:14:02.390918 29836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdc83af3311f4ee5a7ceaa5c90fe9947" candidate_uuid: "d317ee71d7a54e67b3da41f11316744d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a4beea4ae947dcbf9453966598285d" is_pre_election: true
I20260620 19:14:02.391124 29967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdc83af3311f4ee5a7ceaa5c90fe9947" candidate_uuid: "d317ee71d7a54e67b3da41f11316744d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7ea684fba0a468c99a18c3edd39ca7c" is_pre_election: true
I20260620 19:14:02.391216 29836 raft_consensus.cc:2454] T bdc83af3311f4ee5a7ceaa5c90fe9947 P 68a4beea4ae947dcbf9453966598285d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d317ee71d7a54e67b3da41f11316744d in term 0.
I20260620 19:14:02.391382 29967 raft_consensus.cc:2454] T bdc83af3311f4ee5a7ceaa5c90fe9947 P e7ea684fba0a468c99a18c3edd39ca7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d317ee71d7a54e67b3da41f11316744d in term 0.
I20260620 19:14:02.391671 29638 leader_election.cc:302] T bdc83af3311f4ee5a7ceaa5c90fe9947 P d317ee71d7a54e67b3da41f11316744d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68a4beea4ae947dcbf9453966598285d, d317ee71d7a54e67b3da41f11316744d; no voters:
I20260620 19:14:02.391919 30038 raft_consensus.cc:2781] T bdc83af3311f4ee5a7ceaa5c90fe9947 P d317ee71d7a54e67b3da41f11316744d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:14:02.391999 30038 raft_consensus.cc:482] T bdc83af3311f4ee5a7ceaa5c90fe9947 P d317ee71d7a54e67b3da41f11316744d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:14:02.392036 30038 raft_consensus.cc:3037] T bdc83af3311f4ee5a7ceaa5c90fe9947 P d317ee71d7a54e67b3da41f11316744d [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:14:02.393251 30038 raft_consensus.cc:504] T bdc83af3311f4ee5a7ceaa5c90fe9947 P d317ee71d7a54e67b3da41f11316744d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d317ee71d7a54e67b3da41f11316744d" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 42941 } } peers { permanent_uuid: "68a4beea4ae947dcbf9453966598285d" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 32913 } } peers { permanent_uuid: "e7ea684fba0a468c99a18c3edd39ca7c" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 32935 } }
I20260620 19:14:02.393559 30038 leader_election.cc:288] T bdc83af3311f4ee5a7ceaa5c90fe9947 P d317ee71d7a54e67b3da41f11316744d [CANDIDATE]: Term 1 election: Requested vote from peers 68a4beea4ae947dcbf9453966598285d (127.9.155.66:32913), e7ea684fba0a468c99a18c3edd39ca7c (127.9.155.67:32935)
I20260620 19:14:02.393898 29836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdc83af3311f4ee5a7ceaa5c90fe9947" candidate_uuid: "d317ee71d7a54e67b3da41f11316744d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a4beea4ae947dcbf9453966598285d"
I20260620 19:14:02.393939 29967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdc83af3311f4ee5a7ceaa5c90fe9947" candidate_uuid: "d317ee71d7a54e67b3da41f11316744d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7ea684fba0a468c99a18c3edd39ca7c"
I20260620 19:14:02.394038 29967 raft_consensus.cc:3037] T bdc83af3311f4ee5a7ceaa5c90fe9947 P e7ea684fba0a468c99a18c3edd39ca7c [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:14:02.394032 29836 raft_consensus.cc:3037] T bdc83af3311f4ee5a7ceaa5c90fe9947 P 68a4beea4ae947dcbf9453966598285d [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:14:02.395298 29967 raft_consensus.cc:2454] T bdc83af3311f4ee5a7ceaa5c90fe9947 P e7ea684fba0a468c99a18c3edd39ca7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d317ee71d7a54e67b3da41f11316744d in term 1.
I20260620 19:14:02.395319 29836 raft_consensus.cc:2454] T bdc83af3311f4ee5a7ceaa5c90fe9947 P 68a4beea4ae947dcbf9453966598285d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d317ee71d7a54e67b3da41f11316744d in term 1.
I20260620 19:14:02.395584 29638 leader_election.cc:302] T bdc83af3311f4ee5a7ceaa5c90fe9947 P d317ee71d7a54e67b3da41f11316744d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d317ee71d7a54e67b3da41f11316744d, e7ea684fba0a468c99a18c3edd39ca7c; no voters:
I20260620 19:14:02.395746 30038 raft_consensus.cc:2781] T bdc83af3311f4ee5a7ceaa5c90fe9947 P d317ee71d7a54e67b3da41f11316744d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:14:02.395947 30038 raft_consensus.cc:686] T bdc83af3311f4ee5a7ceaa5c90fe9947 P d317ee71d7a54e67b3da41f11316744d [term 1 LEADER]: Becoming Leader. State: Replica: d317ee71d7a54e67b3da41f11316744d, State: Running, Role: LEADER
I20260620 19:14:02.396287 30038 consensus_queue.cc:237] T bdc83af3311f4ee5a7ceaa5c90fe9947 P d317ee71d7a54e67b3da41f11316744d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d317ee71d7a54e67b3da41f11316744d" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 42941 } } peers { permanent_uuid: "68a4beea4ae947dcbf9453966598285d" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 32913 } } peers { permanent_uuid: "e7ea684fba0a468c99a18c3edd39ca7c" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 32935 } }
I20260620 19:14:02.399173 29562 catalog_manager.cc:5495] T bdc83af3311f4ee5a7ceaa5c90fe9947 P d317ee71d7a54e67b3da41f11316744d reported cstate change: term changed from 0 to 1, leader changed from <none> to d317ee71d7a54e67b3da41f11316744d (127.9.155.65). New cstate: current_term: 1 leader_uuid: "d317ee71d7a54e67b3da41f11316744d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d317ee71d7a54e67b3da41f11316744d" member_type: VOTER last_known_addr { host: "127.9.155.65" port: 42941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "68a4beea4ae947dcbf9453966598285d" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 32913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7ea684fba0a468c99a18c3edd39ca7c" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 32935 } health_report { overall_health: UNKNOWN } } }
I20260620 19:14:02.416352 30026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:02.413247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36304 (local address 127.9.155.65:42941)
0620 19:14:02.413355 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:02.413359 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:02.413371 (+ 12us) server_negotiation.cc:407] Connection header received
0620 19:14:02.413436 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:02.413438 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:02.413489 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:02.413574 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:14:02.413783 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:02.414261 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:02.415062 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:02.415257 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:02.415318 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:02.415576 (+ 258us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:14:02.415997 (+ 421us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:14:02.416082 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:02.416160 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 19:14:02.416211 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 19:14:02.422114 29967 raft_consensus.cc:1261] T bdc83af3311f4ee5a7ceaa5c90fe9947 P e7ea684fba0a468c99a18c3edd39ca7c [term 1 FOLLOWER]: Refusing update from remote peer d317ee71d7a54e67b3da41f11316744d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:14:02.422114 29836 raft_consensus.cc:1261] T bdc83af3311f4ee5a7ceaa5c90fe9947 P 68a4beea4ae947dcbf9453966598285d [term 1 FOLLOWER]: Refusing update from remote peer d317ee71d7a54e67b3da41f11316744d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:14:02.422833 30038 consensus_queue.cc:1035] T bdc83af3311f4ee5a7ceaa5c90fe9947 P d317ee71d7a54e67b3da41f11316744d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7ea684fba0a468c99a18c3edd39ca7c" member_type: VOTER last_known_addr { host: "127.9.155.67" port: 32935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:14:02.423043 30040 consensus_queue.cc:1035] T bdc83af3311f4ee5a7ceaa5c90fe9947 P d317ee71d7a54e67b3da41f11316744d [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a4beea4ae947dcbf9453966598285d" member_type: VOTER last_known_addr { host: "127.9.155.66" port: 32913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 19:14:02.430255 29752 tablet.cc:2369] T bdc83af3311f4ee5a7ceaa5c90fe9947 P d317ee71d7a54e67b3da41f11316744d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:14:02.430974 30045 mvcc.cc:205] Tried to move back new op lower bound from 7299001722553913344 to 7299001722457698304. Current Snapshot: MvccSnapshot[applied={T|T < 7299001722553913344}]
I20260620 19:14:02.431115 30044 mvcc.cc:205] Tried to move back new op lower bound from 7299001722553913344 to 7299001722457698304. Current Snapshot: MvccSnapshot[applied={T|T < 7299001722553913344}]
I20260620 19:14:02.440503 29563 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52196:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:14:02.440663 29563 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52196:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:14:02.443051 29563 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ff192e1725064a14854993710ece4abd: Sending DeleteTablet for 3 replicas of tablet bdc83af3311f4ee5a7ceaa5c90fe9947
I20260620 19:14:02.443849 29685 tablet_service.cc:1514] Processing DeleteTablet for tablet bdc83af3311f4ee5a7ceaa5c90fe9947 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:14:02 UTC) from {username='slave'} at 127.0.0.1:36296
I20260620 19:14:02.443979 29816 tablet_service.cc:1514] Processing DeleteTablet for tablet bdc83af3311f4ee5a7ceaa5c90fe9947 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:14:02 UTC) from {username='slave'} at 127.0.0.1:35304
I20260620 19:14:02.444382 30053 tablet_replica.cc:330] T bdc83af3311f4ee5a7ceaa5c90fe9947 P d317ee71d7a54e67b3da41f11316744d: stopping tablet replica
I20260620 19:14:02.444437 29947 tablet_service.cc:1514] Processing DeleteTablet for tablet bdc83af3311f4ee5a7ceaa5c90fe9947 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:14:02 UTC) from {username='slave'} at 127.0.0.1:56206
I20260620 19:14:02.444492 30054 tablet_replica.cc:330] T bdc83af3311f4ee5a7ceaa5c90fe9947 P 68a4beea4ae947dcbf9453966598285d: stopping tablet replica
I20260620 19:14:02.444820 30054 raft_consensus.cc:2229] T bdc83af3311f4ee5a7ceaa5c90fe9947 P 68a4beea4ae947dcbf9453966598285d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:14:02.444821 30053 raft_consensus.cc:2229] T bdc83af3311f4ee5a7ceaa5c90fe9947 P d317ee71d7a54e67b3da41f11316744d [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:14:02.445151 30054 raft_consensus.cc:2258] T bdc83af3311f4ee5a7ceaa5c90fe9947 P 68a4beea4ae947dcbf9453966598285d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:14:02.445345 30053 raft_consensus.cc:2258] T bdc83af3311f4ee5a7ceaa5c90fe9947 P d317ee71d7a54e67b3da41f11316744d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:14:02.445442 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 29623
I20260620 19:14:02.447393 30054 ts_tablet_manager.cc:1905] T bdc83af3311f4ee5a7ceaa5c90fe9947 P 68a4beea4ae947dcbf9453966598285d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:14:02.448436 30055 tablet_replica.cc:330] T bdc83af3311f4ee5a7ceaa5c90fe9947 P e7ea684fba0a468c99a18c3edd39ca7c: stopping tablet replica
I20260620 19:14:02.448761 30055 raft_consensus.cc:2229] T bdc83af3311f4ee5a7ceaa5c90fe9947 P e7ea684fba0a468c99a18c3edd39ca7c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:14:02.449108 30055 raft_consensus.cc:2258] T bdc83af3311f4ee5a7ceaa5c90fe9947 P e7ea684fba0a468c99a18c3edd39ca7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:14:02.450083 30054 ts_tablet_manager.cc:1918] T bdc83af3311f4ee5a7ceaa5c90fe9947 P 68a4beea4ae947dcbf9453966598285d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:14:02.450181 30054 log.cc:1196] T bdc83af3311f4ee5a7ceaa5c90fe9947 P 68a4beea4ae947dcbf9453966598285d: Deleting WAL directory at /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982595231230-9837-0/minicluster-data/ts-1/wal/wals/bdc83af3311f4ee5a7ceaa5c90fe9947
I20260620 19:14:02.450479 30054 ts_tablet_manager.cc:1939] T bdc83af3311f4ee5a7ceaa5c90fe9947 P 68a4beea4ae947dcbf9453966598285d: Deleting consensus metadata
I20260620 19:14:02.450678 30055 ts_tablet_manager.cc:1905] T bdc83af3311f4ee5a7ceaa5c90fe9947 P e7ea684fba0a468c99a18c3edd39ca7c: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 19:14:02.451370 29549 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.155.65:42941 (error 108)
I20260620 19:14:02.451678 29547 catalog_manager.cc:4841] TS 68a4beea4ae947dcbf9453966598285d (127.9.155.66:32913): tablet bdc83af3311f4ee5a7ceaa5c90fe9947 (table test-table [id=88546ffe51a849fab3634cc6bbfa49bc]) successfully deleted
I20260620 19:14:02.452104 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 29754
I20260620 19:14:02.452988 30027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:02.452240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.65:42941 (local address 127.0.0.1:36312)
0620 19:14:02.452857 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:02.452916 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.65:42941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":538}
I20260620 19:14:02.453058 30055 ts_tablet_manager.cc:1918] T bdc83af3311f4ee5a7ceaa5c90fe9947 P e7ea684fba0a468c99a18c3edd39ca7c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:14:02.453145 30055 log.cc:1196] T bdc83af3311f4ee5a7ceaa5c90fe9947 P e7ea684fba0a468c99a18c3edd39ca7c: Deleting WAL directory at /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982595231230-9837-0/minicluster-data/ts-2/wal/wals/bdc83af3311f4ee5a7ceaa5c90fe9947
W20260620 19:14:02.453150 29549 catalog_manager.cc:4584] TS d317ee71d7a54e67b3da41f11316744d (127.9.155.65:42941): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bdc83af3311f4ee5a7ceaa5c90fe9947: Network error: Client connection negotiation failed: client connection to 127.9.155.65:42941: connect: Connection refused (error 111)
I20260620 19:14:02.453426 30055 ts_tablet_manager.cc:1939] T bdc83af3311f4ee5a7ceaa5c90fe9947 P e7ea684fba0a468c99a18c3edd39ca7c: Deleting consensus metadata
I20260620 19:14:02.454370 29547 catalog_manager.cc:4841] TS e7ea684fba0a468c99a18c3edd39ca7c (127.9.155.67:32935): tablet bdc83af3311f4ee5a7ceaa5c90fe9947 (table test-table [id=88546ffe51a849fab3634cc6bbfa49bc]) successfully deleted
I20260620 19:14:02.457647 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 29885
I20260620 19:14:02.462708 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 29532
2026-06-20T19:14:02Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (970 ms)
[----------] 6 tests from EncryptionPolicyTest (9802 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982595231230-9837-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:14:02 dist-test-slave-n3fx krb5kdc[30059](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:14:02 dist-test-slave-n3fx krb5kdc[30059](info): set up 2 sockets
Jun 20 19:14:02 dist-test-slave-n3fx krb5kdc[30059](info): commencing operation
krb5kdc: starting...
W20260620 19:14:03.420353 9837 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982595231230-9837-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982595231230-9837-0/krb5kdc/test-user.keytab.
Jun 20 19:14:03 dist-test-slave-n3fx krb5kdc[30059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:14:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:14:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982595231230-9837-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982595231230-9837-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 19:14:03.537443 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982595231230-9837-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982595231230-9837-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982595231230-9837-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45425
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982595231230-9837-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45425
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:14:03.646505 30072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:14:03.646803 30072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:14:03.646847 30072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:14:03.650978 30072 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:14:03.651052 30072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:14:03.651070 30072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:14:03.651086 30072 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:14:03.651100 30072 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:14:03.656571 30072 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982595231230-9837-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982595231230-9837-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45425
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982595231230-9837-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30072
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982595231230-9837-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:14:03.657661 30072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:14:03.658658 30072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:14:03.667460 30077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:14:03.667472 30078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:14:03.667523 30072 server_base.cc:1034] running on GCE node
W20260620 19:14:03.667472 30080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:14:03.668378 30072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:14:03.669519 30072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:14:03.670701 30072 hybrid_clock.cc:648] HybridClock initialized: now 1781982843670674 us; error 43 us; skew 500 ppm
Jun 20 19:14:03 dist-test-slave-n3fx krb5kdc[30059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982843, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:14:03.677546 30072 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 19:14:03.678593 30072 webserver.cc:460] Webserver started at http://127.0.0.1:39817/ using document root <none> and password file <none>
I20260620 19:14:03.679126 30072 fs_manager.cc:359] Metadata directory not provided
I20260620 19:14:03.679185 30072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:14:03.679360 30072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:14:03.681481 30072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982595231230-9837-0/minicluster-data/master-0/data/instance:
uuid: "62450345f2b74886ae545322160dd5ad"
format_stamp: "Formatted at 2026-06-20 19:14:03 on dist-test-slave-n3fx"
server_key: "e1474f59f0d7881ed3f3b0af83b74b0c"
server_key_iv: "c54fa64ab4b3c14c230548075dc47c39"
server_key_version: "encryptionkey@0"
I20260620 19:14:03.681985 30072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982595231230-9837-0/minicluster-data/master-0/wal/instance:
uuid: "62450345f2b74886ae545322160dd5ad"
format_stamp: "Formatted at 2026-06-20 19:14:03 on dist-test-slave-n3fx"
server_key: "e1474f59f0d7881ed3f3b0af83b74b0c"
server_key_iv: "c54fa64ab4b3c14c230548075dc47c39"
server_key_version: "encryptionkey@0"
I20260620 19:14:03.685716 30072 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 19:14:03.688371 30087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:03.689894 30072 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:14:03.690017 30072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982595231230-9837-0/minicluster-data/master-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982595231230-9837-0/minicluster-data/master-0/wal
uuid: "62450345f2b74886ae545322160dd5ad"
format_stamp: "Formatted at 2026-06-20 19:14:03 on dist-test-slave-n3fx"
server_key: "e1474f59f0d7881ed3f3b0af83b74b0c"
server_key_iv: "c54fa64ab4b3c14c230548075dc47c39"
server_key_version: "encryptionkey@0"
I20260620 19:14:03.690126 30072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982595231230-9837-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982595231230-9837-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982595231230-9837-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:14:03.711072 30072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:14:03.714200 30072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:14:03.714360 30072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:14:03.721365 30072 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45425
I20260620 19:14:03.721429 30139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45425 every 8 connection(s)
I20260620 19:14:03.722354 30072 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
I20260620 19:14:03.723897 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 30072
I20260620 19:14:03.724017 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982595231230-9837-0/minicluster-data/master-0/wal/instance
I20260620 19:14:03.724411 9837 external_mini_cluster.cc:1428] Setting key cb6d6573dafda234f9d99a85a99d6126
I20260620 19:14:03.725584 30140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:14:03.732676 30140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62450345f2b74886ae545322160dd5ad: Bootstrap starting.
Jun 20 19:14:03 dist-test-slave-n3fx krb5kdc[30059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 19:14:03.735610 30140 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62450345f2b74886ae545322160dd5ad: Neither blocks nor log segments found. Creating new log.
I20260620 19:14:03.736491 30140 log.cc:824] T 00000000000000000000000000000000 P 62450345f2b74886ae545322160dd5ad: Log is configured to *not* fsync() on all Append() calls
I20260620 19:14:03.738894 30140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62450345f2b74886ae545322160dd5ad: No bootstrap required, opened a new log
I20260620 19:14:03.739815 30143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:03.725921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46251 (local address 127.0.0.1:45425)
0620 19:14:03.726350 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:03.726369 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:03.726412 (+ 43us) server_negotiation.cc:407] Connection header received
0620 19:14:03.727062 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:03.727080 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:03.727434 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:03.727714 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:03.728688 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:03.729753 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:03.730929 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:03.731434 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:03.731564 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:03.734785 (+ 3221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:03.734792 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:03.734806 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:14:03.734840 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:03.736578 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:03.737168 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:03.737175 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:03.737181 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:03.737301 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:03.737624 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:03.737627 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:03.737629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:03.738129 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:03.738332 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:03.738609 (+ 277us) server_negotiation.cc:299] Negotiation successful
0620 19:14:03.738847 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":129,"threads_started":1}
I20260620 19:14:03.743028 9837 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:14:03.743110 30140 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62450345f2b74886ae545322160dd5ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62450345f2b74886ae545322160dd5ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45425 } }
I20260620 19:14:03.743384 30140 raft_consensus.cc:374] T 00000000000000000000000000000000 P 62450345f2b74886ae545322160dd5ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:14:03.743448 30140 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62450345f2b74886ae545322160dd5ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62450345f2b74886ae545322160dd5ad, State: Initialized, Role: FOLLOWER
I20260620 19:14:03.743935 30140 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62450345f2b74886ae545322160dd5ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62450345f2b74886ae545322160dd5ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45425 } }
I20260620 19:14:03.744097 30140 raft_consensus.cc:388] T 00000000000000000000000000000000 P 62450345f2b74886ae545322160dd5ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:14:03.744174 30140 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62450345f2b74886ae545322160dd5ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:14:03.744295 30140 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62450345f2b74886ae545322160dd5ad [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:14:03.745466 30140 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62450345f2b74886ae545322160dd5ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62450345f2b74886ae545322160dd5ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45425 } }
I20260620 19:14:03.745802 30140 leader_election.cc:302] T 00000000000000000000000000000000 P 62450345f2b74886ae545322160dd5ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 62450345f2b74886ae545322160dd5ad; no voters:
I20260620 19:14:03.746093 30140 leader_election.cc:288] T 00000000000000000000000000000000 P 62450345f2b74886ae545322160dd5ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:14:03.746201 30150 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 62450345f2b74886ae545322160dd5ad [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:14:03.746407 30150 raft_consensus.cc:686] T 00000000000000000000000000000000 P 62450345f2b74886ae545322160dd5ad [term 1 LEADER]: Becoming Leader. State: Replica: 62450345f2b74886ae545322160dd5ad, State: Running, Role: LEADER
I20260620 19:14:03.746820 30150 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62450345f2b74886ae545322160dd5ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62450345f2b74886ae545322160dd5ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45425 } }
I20260620 19:14:03.747427 30140 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62450345f2b74886ae545322160dd5ad [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:14:03.749464 30151 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62450345f2b74886ae545322160dd5ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "62450345f2b74886ae545322160dd5ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62450345f2b74886ae545322160dd5ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45425 } } }
I20260620 19:14:03.749662 30152 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62450345f2b74886ae545322160dd5ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62450345f2b74886ae545322160dd5ad. Latest consensus state: current_term: 1 leader_uuid: "62450345f2b74886ae545322160dd5ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62450345f2b74886ae545322160dd5ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45425 } } }
I20260620 19:14:03.749913 30151 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62450345f2b74886ae545322160dd5ad [sys.catalog]: This master's current role is: LEADER
I20260620 19:14:03.749917 30152 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62450345f2b74886ae545322160dd5ad [sys.catalog]: This master's current role is: LEADER
W20260620 19:14:03.752310 30164 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 62450345f2b74886ae545322160dd5ad: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:14:03.752374 30164 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:14:03.752444 30165 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:14:03.753328 30165 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:14:03.754001 30143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:03.745285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56013 (local address 127.0.0.1:45425)
0620 19:14:03.745411 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:03.745416 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:03.745660 (+ 244us) server_negotiation.cc:407] Connection header received
0620 19:14:03.745706 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:03.745708 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:03.745762 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:03.745823 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:03.746623 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:03.747380 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:03.748459 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:03.748767 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:03.748857 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:03.750453 (+ 1596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:03.750457 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:03.750460 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:14:03.750485 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:03.752006 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:03.752606 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:03.752610 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:03.752611 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:03.752695 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:03.753068 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:03.753071 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:03.753073 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:03.753302 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:03.753452 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:03.753754 (+ 302us) server_negotiation.cc:299] Negotiation successful
0620 19:14:03.753855 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":48}
I20260620 19:14:03.759300 30165 catalog_manager.cc:1347] Generated new cluster ID: 9bacc429919d4f0c96eec319f9786b90
I20260620 19:14:03.759377 30165 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:14:03.784052 30165 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:14:03.785404 30165 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:14:03.797298 30165 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 62450345f2b74886ae545322160dd5ad: Generated new TSK 0
I20260620 19:14:03.798007 30165 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:14:03.807087 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 30072
W20260620 19:14:03.812093 30057 connection.cc:535] client connection to 127.0.0.1:45425 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45425): Connection reset by peer (error 104)
2026-06-20T19:14:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1343 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982595231230-9837-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:14:03 dist-test-slave-n3fx krb5kdc[30171](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:14:03 dist-test-slave-n3fx krb5kdc[30171](info): set up 2 sockets
Jun 20 19:14:03 dist-test-slave-n3fx krb5kdc[30171](info): commencing operation
krb5kdc: starting...
W20260620 19:14:04.768436 9837 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982595231230-9837-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982595231230-9837-0/krb5kdc/test-user.keytab.
Jun 20 19:14:04 dist-test-slave-n3fx krb5kdc[30171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982844, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:14:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:14:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982595231230-9837-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982595231230-9837-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 19:14:04.890993 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982595231230-9837-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982595231230-9837-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982595231230-9837-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40991
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982595231230-9837-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40991
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:14:05.005247 30184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:14:05.005605 30184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:14:05.005671 30184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:14:05.010161 30184 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:14:05.010251 30184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:14:05.010281 30184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:14:05.010304 30184 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:14:05.010324 30184 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:14:05.016083 30184 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982595231230-9837-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982595231230-9837-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40991
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982595231230-9837-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30184
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982595231230-9837-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:14:05.017297 30184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:14:05.018359 30184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:14:05.027190 30190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:14:05.027251 30192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:14:05.027359 30184 server_base.cc:1034] running on GCE node
W20260620 19:14:05.027160 30189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:14:05.028038 30184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:14:05.029254 30184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:14:05.030445 30184 hybrid_clock.cc:648] HybridClock initialized: now 1781982845030418 us; error 41 us; skew 500 ppm
Jun 20 19:14:05 dist-test-slave-n3fx krb5kdc[30171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982845, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:14:05.033313 30184 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 19:14:05.034204 30184 webserver.cc:460] Webserver started at http://127.0.0.1:36961/ using document root <none> and password file <none>
I20260620 19:14:05.034741 30184 fs_manager.cc:359] Metadata directory not provided
I20260620 19:14:05.034811 30184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:14:05.034979 30184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:14:05.037173 30184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982595231230-9837-0/minicluster-data/master-0/data/instance:
uuid: "85c9f28bb5394cf5a40e0e360ed56fd4"
format_stamp: "Formatted at 2026-06-20 19:14:05 on dist-test-slave-n3fx"
server_key: "8fa3c5aaf3e132a09916f70ce75336d8"
server_key_iv: "c24538a6b1fc90ea3410f2962628847b"
server_key_version: "encryptionkey@0"
I20260620 19:14:05.037727 30184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982595231230-9837-0/minicluster-data/master-0/wal/instance:
uuid: "85c9f28bb5394cf5a40e0e360ed56fd4"
format_stamp: "Formatted at 2026-06-20 19:14:05 on dist-test-slave-n3fx"
server_key: "8fa3c5aaf3e132a09916f70ce75336d8"
server_key_iv: "c24538a6b1fc90ea3410f2962628847b"
server_key_version: "encryptionkey@0"
I20260620 19:14:05.041780 30184 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 19:14:05.044744 30199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:05.046214 30184 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 19:14:05.046353 30184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982595231230-9837-0/minicluster-data/master-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982595231230-9837-0/minicluster-data/master-0/wal
uuid: "85c9f28bb5394cf5a40e0e360ed56fd4"
format_stamp: "Formatted at 2026-06-20 19:14:05 on dist-test-slave-n3fx"
server_key: "8fa3c5aaf3e132a09916f70ce75336d8"
server_key_iv: "c24538a6b1fc90ea3410f2962628847b"
server_key_version: "encryptionkey@0"
I20260620 19:14:05.046491 30184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982595231230-9837-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982595231230-9837-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982595231230-9837-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:14:05.061218 30184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:14:05.064715 30184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:14:05.064913 30184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:14:05.072155 30184 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40991
I20260620 19:14:05.072175 30251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40991 every 8 connection(s)
I20260620 19:14:05.073139 30184 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
I20260620 19:14:05.075985 30252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:14:05.077576 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 30184
I20260620 19:14:05.077695 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982595231230-9837-0/minicluster-data/master-0/wal/instance
I20260620 19:14:05.077973 9837 external_mini_cluster.cc:1428] Setting key a589ef80d9cb188ab33cdd26cd791cf2
I20260620 19:14:05.082777 30252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85c9f28bb5394cf5a40e0e360ed56fd4: Bootstrap starting.
I20260620 19:14:05.085469 30252 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85c9f28bb5394cf5a40e0e360ed56fd4: Neither blocks nor log segments found. Creating new log.
I20260620 19:14:05.086376 30252 log.cc:824] T 00000000000000000000000000000000 P 85c9f28bb5394cf5a40e0e360ed56fd4: Log is configured to *not* fsync() on all Append() calls
I20260620 19:14:05.088513 30255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:05.079397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37789 (local address 127.0.0.1:40991)
0620 19:14:05.079891 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:05.079909 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:05.079952 (+ 43us) server_negotiation.cc:407] Connection header received
0620 19:14:05.080787 (+ 835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:05.080813 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:05.081323 (+ 510us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:05.081724 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:05.082171 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:05.083548 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:05.084557 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:05.085230 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:05.085402 (+ 172us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:05.086407 (+ 1005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:05.086413 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:05.086431 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:14:05.086934 (+ 503us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:05.086992 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:14:05.087016 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:05.087137 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:05.087352 (+ 215us) server_negotiation.cc:299] Negotiation successful
0620 19:14:05.087498 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":153,"threads_started":1}
I20260620 19:14:05.088884 30252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85c9f28bb5394cf5a40e0e360ed56fd4: No bootstrap required, opened a new log
I20260620 19:14:05.091498 9837 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:14:05.092645 30252 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85c9f28bb5394cf5a40e0e360ed56fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c9f28bb5394cf5a40e0e360ed56fd4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40991 } }
I20260620 19:14:05.092937 30252 raft_consensus.cc:374] T 00000000000000000000000000000000 P 85c9f28bb5394cf5a40e0e360ed56fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:14:05.093006 30252 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85c9f28bb5394cf5a40e0e360ed56fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85c9f28bb5394cf5a40e0e360ed56fd4, State: Initialized, Role: FOLLOWER
I20260620 19:14:05.093493 30252 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85c9f28bb5394cf5a40e0e360ed56fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c9f28bb5394cf5a40e0e360ed56fd4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40991 } }
I20260620 19:14:05.093670 30252 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85c9f28bb5394cf5a40e0e360ed56fd4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:14:05.093791 30252 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85c9f28bb5394cf5a40e0e360ed56fd4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:14:05.093902 30252 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85c9f28bb5394cf5a40e0e360ed56fd4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:14:05.095141 30252 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85c9f28bb5394cf5a40e0e360ed56fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c9f28bb5394cf5a40e0e360ed56fd4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40991 } }
I20260620 19:14:05.095530 30252 leader_election.cc:302] T 00000000000000000000000000000000 P 85c9f28bb5394cf5a40e0e360ed56fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85c9f28bb5394cf5a40e0e360ed56fd4; no voters:
I20260620 19:14:05.095865 30252 leader_election.cc:288] T 00000000000000000000000000000000 P 85c9f28bb5394cf5a40e0e360ed56fd4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:14:05.095955 30262 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85c9f28bb5394cf5a40e0e360ed56fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:14:05.096124 30262 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85c9f28bb5394cf5a40e0e360ed56fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 85c9f28bb5394cf5a40e0e360ed56fd4, State: Running, Role: LEADER
I20260620 19:14:05.096530 30262 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85c9f28bb5394cf5a40e0e360ed56fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c9f28bb5394cf5a40e0e360ed56fd4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40991 } }
I20260620 19:14:05.097289 30252 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85c9f28bb5394cf5a40e0e360ed56fd4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:14:05.098157 30255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:05.093666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37781 (local address 127.0.0.1:40991)
0620 19:14:05.093787 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:05.093791 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:05.093937 (+ 146us) server_negotiation.cc:407] Connection header received
0620 19:14:05.094086 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:05.094089 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:05.094140 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:05.094208 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:05.094452 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:05.095182 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:05.096292 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:05.096580 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:05.096648 (+ 68us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:05.097653 (+ 1005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:05.097656 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:05.097658 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:14:05.097672 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:05.097698 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:14:05.097712 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:05.097806 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:05.097962 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 19:14:05.098007 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 19:14:05.098196 30263 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85c9f28bb5394cf5a40e0e360ed56fd4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "85c9f28bb5394cf5a40e0e360ed56fd4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c9f28bb5394cf5a40e0e360ed56fd4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40991 } } }
I20260620 19:14:05.098552 30263 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85c9f28bb5394cf5a40e0e360ed56fd4 [sys.catalog]: This master's current role is: LEADER
I20260620 19:14:05.098453 30264 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85c9f28bb5394cf5a40e0e360ed56fd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85c9f28bb5394cf5a40e0e360ed56fd4. Latest consensus state: current_term: 1 leader_uuid: "85c9f28bb5394cf5a40e0e360ed56fd4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c9f28bb5394cf5a40e0e360ed56fd4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40991 } } }
I20260620 19:14:05.098856 30264 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85c9f28bb5394cf5a40e0e360ed56fd4 [sys.catalog]: This master's current role is: LEADER
I20260620 19:14:05.099143 30271 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:14:05.102522 30271 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:14:05.108423 30271 catalog_manager.cc:1347] Generated new cluster ID: c70722b34e5f4153a5e99b3fccd3e855
I20260620 19:14:05.108505 30271 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:14:05.140245 30271 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:14:05.141674 30271 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:14:05.149654 30271 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 85c9f28bb5394cf5a40e0e360ed56fd4: Generated new TSK 0
I20260620 19:14:05.150559 30271 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:14:05.155046 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 30184
W20260620 19:14:05.159878 30169 connection.cc:535] client connection to 127.0.0.1:40991 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40991): Connection reset by peer (error 104)
2026-06-20T19:14:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1348 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982595231230-9837-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:14:05 dist-test-slave-n3fx krb5kdc[30283](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:14:05 dist-test-slave-n3fx krb5kdc[30283](info): set up 2 sockets
Jun 20 19:14:05 dist-test-slave-n3fx krb5kdc[30283](info): commencing operation
krb5kdc: starting...
W20260620 19:14:06.112155 9837 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982595231230-9837-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982595231230-9837-0/krb5kdc/test-user.keytab.
Jun 20 19:14:06 dist-test-slave-n3fx krb5kdc[30283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982846, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:14:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:14:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982595231230-9837-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982595231230-9837-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 19:14:06.231506 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982595231230-9837-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982595231230-9837-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982595231230-9837-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34401
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982595231230-9837-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34401
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:14:06.340754 30296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:14:06.341030 30296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:14:06.341090 30296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:14:06.345183 30296 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:14:06.345260 30296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:14:06.345286 30296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:14:06.345310 30296 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:14:06.345340 30296 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:14:06.350687 30296 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982595231230-9837-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982595231230-9837-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34401
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982595231230-9837-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30296
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982595231230-9837-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:14:06.351795 30296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:14:06.352885 30296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 19:14:06.361662 30296 server_base.cc:1034] running on GCE node
W20260620 19:14:06.361608 30302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:14:06.361604 30301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:14:06.361864 30304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:14:06.362450 30296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:14:06.363559 30296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:14:06.364775 30296 hybrid_clock.cc:648] HybridClock initialized: now 1781982846364749 us; error 38 us; skew 500 ppm
Jun 20 19:14:06 dist-test-slave-n3fx krb5kdc[30283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982846, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:14:06.367491 30296 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 19:14:06.368440 30296 webserver.cc:460] Webserver started at http://127.0.0.1:46735/ using document root <none> and password file <none>
I20260620 19:14:06.368986 30296 fs_manager.cc:359] Metadata directory not provided
I20260620 19:14:06.369055 30296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:14:06.369251 30296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:14:06.371295 30296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982595231230-9837-0/minicluster-data/master-0/data/instance:
uuid: "d29f44ad398140318c70bb5079c8b081"
format_stamp: "Formatted at 2026-06-20 19:14:06 on dist-test-slave-n3fx"
server_key: "2333fba67608da3a4b05a2ccea563fee"
server_key_iv: "0325518056790a40c13eda396a61dea3"
server_key_version: "encryptionkey@0"
I20260620 19:14:06.371830 30296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982595231230-9837-0/minicluster-data/master-0/wal/instance:
uuid: "d29f44ad398140318c70bb5079c8b081"
format_stamp: "Formatted at 2026-06-20 19:14:06 on dist-test-slave-n3fx"
server_key: "2333fba67608da3a4b05a2ccea563fee"
server_key_iv: "0325518056790a40c13eda396a61dea3"
server_key_version: "encryptionkey@0"
I20260620 19:14:06.375463 30296 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 19:14:06.377945 30311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:06.379132 30296 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 19:14:06.379245 30296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982595231230-9837-0/minicluster-data/master-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982595231230-9837-0/minicluster-data/master-0/wal
uuid: "d29f44ad398140318c70bb5079c8b081"
format_stamp: "Formatted at 2026-06-20 19:14:06 on dist-test-slave-n3fx"
server_key: "2333fba67608da3a4b05a2ccea563fee"
server_key_iv: "0325518056790a40c13eda396a61dea3"
server_key_version: "encryptionkey@0"
I20260620 19:14:06.379346 30296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982595231230-9837-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982595231230-9837-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982595231230-9837-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:14:06.383905 30296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:14:06.386875 30296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:14:06.387048 30296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:14:06.393769 30296 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34401
I20260620 19:14:06.393772 30363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34401 every 8 connection(s)
I20260620 19:14:06.394721 30296 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
I20260620 19:14:06.397630 30364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:14:06.397610 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 30296
I20260620 19:14:06.397768 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982595231230-9837-0/minicluster-data/master-0/wal/instance
I20260620 19:14:06.398093 9837 external_mini_cluster.cc:1428] Setting key 0919d18c5c22f010612f88e6c07c15c4
I20260620 19:14:06.403687 30367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:06.399473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46075 (local address 127.0.0.1:34401)
0620 19:14:06.399904 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:06.399920 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:06.399959 (+ 39us) server_negotiation.cc:407] Connection header received
0620 19:14:06.400729 (+ 770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:06.400750 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:06.401070 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:06.401405 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:06.402361 (+ 956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:06.402366 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:06.402378 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:14:06.402389 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:06.402448 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:14:06.402468 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:06.402605 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:06.402689 (+ 84us) server_negotiation.cc:299] Negotiation successful
0620 19:14:06.402917 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":120,"threads_started":1}
I20260620 19:14:06.406050 30364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d29f44ad398140318c70bb5079c8b081: Bootstrap starting.
I20260620 19:14:06.406785 9837 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:14:06.408533 30364 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d29f44ad398140318c70bb5079c8b081: Neither blocks nor log segments found. Creating new log.
I20260620 19:14:06.409447 30364 log.cc:824] T 00000000000000000000000000000000 P d29f44ad398140318c70bb5079c8b081: Log is configured to *not* fsync() on all Append() calls
I20260620 19:14:06.410493 30367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:06.408800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59437 (local address 127.0.0.1:34401)
0620 19:14:06.408940 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:06.408946 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:06.409138 (+ 192us) server_negotiation.cc:407] Connection header received
0620 19:14:06.409240 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:06.409243 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:06.409279 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:06.409347 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:06.410067 (+ 720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:06.410070 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:06.410073 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:14:06.410083 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:06.410111 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:14:06.410123 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:06.410282 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:06.410344 (+ 62us) server_negotiation.cc:299] Negotiation successful
0620 19:14:06.410379 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 19:14:06.411767 30364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d29f44ad398140318c70bb5079c8b081: No bootstrap required, opened a new log
I20260620 19:14:06.415477 30364 raft_consensus.cc:348] T 00000000000000000000000000000000 P d29f44ad398140318c70bb5079c8b081 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d29f44ad398140318c70bb5079c8b081" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34401 } }
I20260620 19:14:06.415727 30364 raft_consensus.cc:374] T 00000000000000000000000000000000 P d29f44ad398140318c70bb5079c8b081 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:14:06.415787 30364 raft_consensus.cc:729] T 00000000000000000000000000000000 P d29f44ad398140318c70bb5079c8b081 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d29f44ad398140318c70bb5079c8b081, State: Initialized, Role: FOLLOWER
I20260620 19:14:06.416249 30364 consensus_queue.cc:260] T 00000000000000000000000000000000 P d29f44ad398140318c70bb5079c8b081 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d29f44ad398140318c70bb5079c8b081" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34401 } }
I20260620 19:14:06.416407 30364 raft_consensus.cc:388] T 00000000000000000000000000000000 P d29f44ad398140318c70bb5079c8b081 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:14:06.416476 30364 raft_consensus.cc:482] T 00000000000000000000000000000000 P d29f44ad398140318c70bb5079c8b081 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:14:06.416571 30364 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d29f44ad398140318c70bb5079c8b081 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:14:06.417517 30364 raft_consensus.cc:504] T 00000000000000000000000000000000 P d29f44ad398140318c70bb5079c8b081 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d29f44ad398140318c70bb5079c8b081" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34401 } }
I20260620 19:14:06.417866 30364 leader_election.cc:302] T 00000000000000000000000000000000 P d29f44ad398140318c70bb5079c8b081 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d29f44ad398140318c70bb5079c8b081; no voters:
I20260620 19:14:06.418172 30364 leader_election.cc:288] T 00000000000000000000000000000000 P d29f44ad398140318c70bb5079c8b081 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:14:06.418339 30374 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d29f44ad398140318c70bb5079c8b081 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:14:06.418545 30374 raft_consensus.cc:686] T 00000000000000000000000000000000 P d29f44ad398140318c70bb5079c8b081 [term 1 LEADER]: Becoming Leader. State: Replica: d29f44ad398140318c70bb5079c8b081, State: Running, Role: LEADER
I20260620 19:14:06.419183 30374 consensus_queue.cc:237] T 00000000000000000000000000000000 P d29f44ad398140318c70bb5079c8b081 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d29f44ad398140318c70bb5079c8b081" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34401 } }
I20260620 19:14:06.419529 30364 sys_catalog.cc:564] T 00000000000000000000000000000000 P d29f44ad398140318c70bb5079c8b081 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:14:06.420812 30375 sys_catalog.cc:455] T 00000000000000000000000000000000 P d29f44ad398140318c70bb5079c8b081 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d29f44ad398140318c70bb5079c8b081" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d29f44ad398140318c70bb5079c8b081" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34401 } } }
I20260620 19:14:06.420928 30375 sys_catalog.cc:458] T 00000000000000000000000000000000 P d29f44ad398140318c70bb5079c8b081 [sys.catalog]: This master's current role is: LEADER
I20260620 19:14:06.421247 30376 sys_catalog.cc:455] T 00000000000000000000000000000000 P d29f44ad398140318c70bb5079c8b081 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d29f44ad398140318c70bb5079c8b081. Latest consensus state: current_term: 1 leader_uuid: "d29f44ad398140318c70bb5079c8b081" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d29f44ad398140318c70bb5079c8b081" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34401 } } }
I20260620 19:14:06.421339 30376 sys_catalog.cc:458] T 00000000000000000000000000000000 P d29f44ad398140318c70bb5079c8b081 [sys.catalog]: This master's current role is: LEADER
W20260620 19:14:06.424124 30389 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d29f44ad398140318c70bb5079c8b081: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:14:06.424206 30389 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:14:06.424311 30384 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:14:06.425024 30384 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:14:06.430819 30384 catalog_manager.cc:1347] Generated new cluster ID: 49921642254744c5b73fb03af90973bd
I20260620 19:14:06.430907 30384 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:14:06.453377 30384 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:14:06.453684 30384 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:14:06.464623 30384 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d29f44ad398140318c70bb5079c8b081: Generated new TSK 0
I20260620 19:14:06.465339 30384 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:14:06.475368 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 30296
2026-06-20T19:14:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1319 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982595231230-9837-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:14:06 dist-test-slave-n3fx krb5kdc[30395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:14:06 dist-test-slave-n3fx krb5kdc[30395](info): set up 2 sockets
Jun 20 19:14:06 dist-test-slave-n3fx krb5kdc[30395](info): commencing operation
krb5kdc: starting...
W20260620 19:14:07.436124 9837 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982595231230-9837-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982595231230-9837-0/krb5kdc/test-user.keytab.
Jun 20 19:14:07 dist-test-slave-n3fx krb5kdc[30395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982847, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:14:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:14:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982595231230-9837-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982595231230-9837-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 19:14:07.555352 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982595231230-9837-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982595231230-9837-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982595231230-9837-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42101
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982595231230-9837-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42101
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:14:07.664706 30408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:14:07.665025 30408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:14:07.665089 30408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:14:07.669299 30408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:14:07.669389 30408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:14:07.669418 30408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:14:07.669438 30408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:14:07.669457 30408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:14:07.675136 30408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982595231230-9837-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982595231230-9837-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42101
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982595231230-9837-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30408
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982595231230-9837-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:14:07.676302 30408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:14:07.677373 30408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:14:07.685847 30416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:14:07.685979 30408 server_base.cc:1034] running on GCE node
W20260620 19:14:07.685892 30414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:14:07.685917 30413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:14:07.686726 30408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:14:07.687904 30408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:14:07.689070 30408 hybrid_clock.cc:648] HybridClock initialized: now 1781982847689048 us; error 40 us; skew 500 ppm
Jun 20 19:14:07 dist-test-slave-n3fx krb5kdc[30395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982847, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:14:07.691805 30408 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 19:14:07.692780 30408 webserver.cc:460] Webserver started at http://127.0.0.1:35657/ using document root <none> and password file <none>
I20260620 19:14:07.693344 30408 fs_manager.cc:359] Metadata directory not provided
I20260620 19:14:07.693411 30408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:14:07.693615 30408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:14:07.695624 30408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982595231230-9837-0/minicluster-data/master-0/data/instance:
uuid: "9f93a51b5a75492fa7fe56b279be616e"
format_stamp: "Formatted at 2026-06-20 19:14:07 on dist-test-slave-n3fx"
server_key: "c742710ec74fec60b752775bf763bc0b"
server_key_iv: "61e491e17619a0ece011e40d0ee7695d"
server_key_version: "encryptionkey@0"
I20260620 19:14:07.696162 30408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982595231230-9837-0/minicluster-data/master-0/wal/instance:
uuid: "9f93a51b5a75492fa7fe56b279be616e"
format_stamp: "Formatted at 2026-06-20 19:14:07 on dist-test-slave-n3fx"
server_key: "c742710ec74fec60b752775bf763bc0b"
server_key_iv: "61e491e17619a0ece011e40d0ee7695d"
server_key_version: "encryptionkey@0"
I20260620 19:14:07.699815 30408 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:14:07.702615 30423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:07.704150 30408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 19:14:07.704363 30408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982595231230-9837-0/minicluster-data/master-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982595231230-9837-0/minicluster-data/master-0/wal
uuid: "9f93a51b5a75492fa7fe56b279be616e"
format_stamp: "Formatted at 2026-06-20 19:14:07 on dist-test-slave-n3fx"
server_key: "c742710ec74fec60b752775bf763bc0b"
server_key_iv: "61e491e17619a0ece011e40d0ee7695d"
server_key_version: "encryptionkey@0"
I20260620 19:14:07.704499 30408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982595231230-9837-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982595231230-9837-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982595231230-9837-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:14:07.721692 30408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:14:07.725582 30408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:14:07.725747 30408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:14:07.732455 30408 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42101
I20260620 19:14:07.732519 30475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42101 every 8 connection(s)
I20260620 19:14:07.733414 30408 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
I20260620 19:14:07.736124 30476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:14:07.741580 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 30408
I20260620 19:14:07.741693 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982595231230-9837-0/minicluster-data/master-0/wal/instance
I20260620 19:14:07.741767 30476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f93a51b5a75492fa7fe56b279be616e: Bootstrap starting.
I20260620 19:14:07.742002 9837 external_mini_cluster.cc:1428] Setting key ed685b24ed65c64a9d785d71dd499621
I20260620 19:14:07.744508 30476 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f93a51b5a75492fa7fe56b279be616e: Neither blocks nor log segments found. Creating new log.
I20260620 19:14:07.745411 30476 log.cc:824] T 00000000000000000000000000000000 P 9f93a51b5a75492fa7fe56b279be616e: Log is configured to *not* fsync() on all Append() calls
I20260620 19:14:07.747852 30476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f93a51b5a75492fa7fe56b279be616e: No bootstrap required, opened a new log
I20260620 19:14:07.752080 30476 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f93a51b5a75492fa7fe56b279be616e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f93a51b5a75492fa7fe56b279be616e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42101 } }
Jun 20 19:14:07 dist-test-slave-n3fx krb5kdc[30395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982847, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 19:14:07.752440 30476 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f93a51b5a75492fa7fe56b279be616e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:14:07.752503 30476 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f93a51b5a75492fa7fe56b279be616e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f93a51b5a75492fa7fe56b279be616e, State: Initialized, Role: FOLLOWER
I20260620 19:14:07.753046 30476 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f93a51b5a75492fa7fe56b279be616e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f93a51b5a75492fa7fe56b279be616e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42101 } }
I20260620 19:14:07.753188 30476 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f93a51b5a75492fa7fe56b279be616e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:14:07.753232 30476 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f93a51b5a75492fa7fe56b279be616e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:14:07.753322 30476 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f93a51b5a75492fa7fe56b279be616e [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:14:07.754447 30476 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f93a51b5a75492fa7fe56b279be616e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f93a51b5a75492fa7fe56b279be616e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42101 } }
I20260620 19:14:07.754777 30476 leader_election.cc:302] T 00000000000000000000000000000000 P 9f93a51b5a75492fa7fe56b279be616e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f93a51b5a75492fa7fe56b279be616e; no voters:
I20260620 19:14:07.755079 30476 leader_election.cc:288] T 00000000000000000000000000000000 P 9f93a51b5a75492fa7fe56b279be616e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:14:07.755164 30481 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f93a51b5a75492fa7fe56b279be616e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:14:07.755330 30481 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f93a51b5a75492fa7fe56b279be616e [term 1 LEADER]: Becoming Leader. State: Replica: 9f93a51b5a75492fa7fe56b279be616e, State: Running, Role: LEADER
I20260620 19:14:07.755678 30481 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f93a51b5a75492fa7fe56b279be616e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f93a51b5a75492fa7fe56b279be616e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42101 } }
I20260620 19:14:07.756456 30476 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f93a51b5a75492fa7fe56b279be616e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:14:07.757336 30483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f93a51b5a75492fa7fe56b279be616e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f93a51b5a75492fa7fe56b279be616e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f93a51b5a75492fa7fe56b279be616e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42101 } } }
I20260620 19:14:07.757464 30483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f93a51b5a75492fa7fe56b279be616e [sys.catalog]: This master's current role is: LEADER
I20260620 19:14:07.757959 30482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f93a51b5a75492fa7fe56b279be616e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f93a51b5a75492fa7fe56b279be616e. Latest consensus state: current_term: 1 leader_uuid: "9f93a51b5a75492fa7fe56b279be616e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f93a51b5a75492fa7fe56b279be616e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42101 } } }
I20260620 19:14:07.758666 30482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f93a51b5a75492fa7fe56b279be616e [sys.catalog]: This master's current role is: LEADER
I20260620 19:14:07.758965 30479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:07.743392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44539 (local address 127.0.0.1:42101)
0620 19:14:07.743885 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:07.743905 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:07.743948 (+ 43us) server_negotiation.cc:407] Connection header received
0620 19:14:07.744848 (+ 900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:07.744874 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:07.745319 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:07.745688 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:07.746594 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:07.748096 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:07.749142 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:07.749891 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:07.750058 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:07.753126 (+ 3068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:07.753132 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:07.753146 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:14:07.753261 (+ 115us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:07.755253 (+ 1992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:07.755731 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:07.755737 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:07.755743 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:07.755884 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:07.756169 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:07.756172 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:07.756174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:07.756741 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:07.756929 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:07.757687 (+ 758us) server_negotiation.cc:299] Negotiation successful
0620 19:14:07.757986 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":161,"threads_started":1}
W20260620 19:14:07.761597 30497 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9f93a51b5a75492fa7fe56b279be616e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:14:07.761678 30497 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:14:07.761765 30490 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:14:07.761909 9837 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:14:07.762614 30490 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:14:07.769162 30490 catalog_manager.cc:1347] Generated new cluster ID: 4c96a07e45d44e34ac5cb979b9a846cc
I20260620 19:14:07.769240 30490 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:14:07.775563 30479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:07.766710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43673 (local address 127.0.0.1:42101)
0620 19:14:07.766839 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:07.766843 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:07.767144 (+ 301us) server_negotiation.cc:407] Connection header received
0620 19:14:07.767303 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:07.767306 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:07.767361 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:07.767452 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:07.768149 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:07.768717 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:07.769726 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:07.770108 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:07.770217 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:07.771758 (+ 1541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:07.771762 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:07.771765 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:14:07.771791 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:07.773384 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:07.773834 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:07.773838 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:07.773840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:07.773921 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:07.774170 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:07.774173 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:07.774175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:07.774389 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:07.774520 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:07.774741 (+ 221us) server_negotiation.cc:299] Negotiation successful
0620 19:14:07.774870 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":40}
I20260620 19:14:07.776255 30490 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:14:07.777263 30490 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:14:07.785000 30490 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f93a51b5a75492fa7fe56b279be616e: Generated new TSK 0
I20260620 19:14:07.785678 30490 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:14:07.790185 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 30408
2026-06-20T19:14:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1314 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982595231230-9837-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:14:07 dist-test-slave-n3fx krb5kdc[30507](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:14:07 dist-test-slave-n3fx krb5kdc[30507](info): set up 2 sockets
Jun 20 19:14:07 dist-test-slave-n3fx krb5kdc[30507](info): commencing operation
krb5kdc: starting...
W20260620 19:14:08.752373 9837 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982595231230-9837-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982595231230-9837-0/krb5kdc/test-user.keytab.
Jun 20 19:14:08 dist-test-slave-n3fx krb5kdc[30507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982848, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:14:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:14:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982595231230-9837-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982595231230-9837-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 19:14:08.871722 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982595231230-9837-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982595231230-9837-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982595231230-9837-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40617
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982595231230-9837-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40617
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:14:08.980973 30520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:14:08.981264 30520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:14:08.981325 30520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:14:08.985479 30520 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:14:08.985562 30520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:14:08.985590 30520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:14:08.985613 30520 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:14:08.985638 30520 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:14:08.991233 30520 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982595231230-9837-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982595231230-9837-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40617
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982595231230-9837-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982595231230-9837-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:14:08.992497 30520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:14:08.993609 30520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 19:14:09.002771 30520 server_base.cc:1034] running on GCE node
W20260620 19:14:09.002754 30526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:14:09.002707 30525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:14:09.002712 30528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:14:09.003618 30520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:14:09.004860 30520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:14:09.006065 30520 hybrid_clock.cc:648] HybridClock initialized: now 1781982849006040 us; error 44 us; skew 500 ppm
Jun 20 19:14:09 dist-test-slave-n3fx krb5kdc[30507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982849, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:14:09.012877 30520 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 19:14:09.013875 30520 webserver.cc:460] Webserver started at http://127.0.0.1:36531/ using document root <none> and password file <none>
I20260620 19:14:09.014438 30520 fs_manager.cc:359] Metadata directory not provided
I20260620 19:14:09.014499 30520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:14:09.014690 30520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:14:09.016876 30520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982595231230-9837-0/minicluster-data/master-0/data/instance:
uuid: "6b85179a4d754da3945a6658820dbf66"
format_stamp: "Formatted at 2026-06-20 19:14:09 on dist-test-slave-n3fx"
server_key: "19acb98d8af58e0fd8bfcd9fa04d03cb"
server_key_iv: "bb1262e8836af57845289c64e14af0d0"
server_key_version: "encryptionkey@0"
I20260620 19:14:09.017424 30520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982595231230-9837-0/minicluster-data/master-0/wal/instance:
uuid: "6b85179a4d754da3945a6658820dbf66"
format_stamp: "Formatted at 2026-06-20 19:14:09 on dist-test-slave-n3fx"
server_key: "19acb98d8af58e0fd8bfcd9fa04d03cb"
server_key_iv: "bb1262e8836af57845289c64e14af0d0"
server_key_version: "encryptionkey@0"
I20260620 19:14:09.021198 30520 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 19:14:09.023551 30535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:09.024796 30520 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:14:09.024920 30520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982595231230-9837-0/minicluster-data/master-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982595231230-9837-0/minicluster-data/master-0/wal
uuid: "6b85179a4d754da3945a6658820dbf66"
format_stamp: "Formatted at 2026-06-20 19:14:09 on dist-test-slave-n3fx"
server_key: "19acb98d8af58e0fd8bfcd9fa04d03cb"
server_key_iv: "bb1262e8836af57845289c64e14af0d0"
server_key_version: "encryptionkey@0"
I20260620 19:14:09.025024 30520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982595231230-9837-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982595231230-9837-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982595231230-9837-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:14:09.043270 30520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:14:09.046643 30520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:14:09.046842 30520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:14:09.053794 30520 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40617
I20260620 19:14:09.053855 30587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40617 every 8 connection(s)
I20260620 19:14:09.054754 30520 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
I20260620 19:14:09.057742 30588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:14:09.058228 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 30520
I20260620 19:14:09.058357 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982595231230-9837-0/minicluster-data/master-0/wal/instance
I20260620 19:14:09.058621 9837 external_mini_cluster.cc:1428] Setting key 338693a7a0dfa425f295e7b58a6729e1
I20260620 19:14:09.064445 30588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b85179a4d754da3945a6658820dbf66: Bootstrap starting.
I20260620 19:14:09.066862 30588 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b85179a4d754da3945a6658820dbf66: Neither blocks nor log segments found. Creating new log.
I20260620 19:14:09.067484 30591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:09.059975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59343 (local address 127.0.0.1:40617)
0620 19:14:09.060448 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:09.060463 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:09.060501 (+ 38us) server_negotiation.cc:407] Connection header received
0620 19:14:09.061182 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:09.061200 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:09.061557 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:09.061903 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:09.062292 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:09.063258 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:09.064144 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:09.064694 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:09.064848 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:09.065709 (+ 861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:09.065713 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:09.065759 (+ 46us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:14:09.066164 (+ 405us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:09.066223 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:14:09.066254 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:09.066348 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:09.066493 (+ 145us) server_negotiation.cc:299] Negotiation successful
0620 19:14:09.066700 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":144,"threads_started":1}
I20260620 19:14:09.067644 30588 log.cc:824] T 00000000000000000000000000000000 P 6b85179a4d754da3945a6658820dbf66: Log is configured to *not* fsync() on all Append() calls
I20260620 19:14:09.069722 30588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b85179a4d754da3945a6658820dbf66: No bootstrap required, opened a new log
I20260620 19:14:09.070323 9837 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:14:09.073477 30588 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b85179a4d754da3945a6658820dbf66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b85179a4d754da3945a6658820dbf66" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40617 } }
I20260620 19:14:09.073709 30588 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b85179a4d754da3945a6658820dbf66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:14:09.073798 30588 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b85179a4d754da3945a6658820dbf66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b85179a4d754da3945a6658820dbf66, State: Initialized, Role: FOLLOWER
I20260620 19:14:09.074215 30588 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b85179a4d754da3945a6658820dbf66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b85179a4d754da3945a6658820dbf66" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40617 } }
I20260620 19:14:09.074361 30588 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b85179a4d754da3945a6658820dbf66 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:14:09.074422 30588 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b85179a4d754da3945a6658820dbf66 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:14:09.074522 30588 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b85179a4d754da3945a6658820dbf66 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:14:09.075523 30588 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b85179a4d754da3945a6658820dbf66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b85179a4d754da3945a6658820dbf66" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40617 } }
I20260620 19:14:09.075884 30588 leader_election.cc:302] T 00000000000000000000000000000000 P 6b85179a4d754da3945a6658820dbf66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b85179a4d754da3945a6658820dbf66; no voters:
I20260620 19:14:09.076089 30591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:09.072343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54131 (local address 127.0.0.1:40617)
0620 19:14:09.072451 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:09.072454 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:09.072668 (+ 214us) server_negotiation.cc:407] Connection header received
0620 19:14:09.072793 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:09.072796 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:09.072842 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:09.072909 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:09.073163 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:09.073683 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:09.074572 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:09.074840 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:09.074925 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:09.075666 (+ 741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:09.075670 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:09.075673 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:14:09.075685 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:09.075712 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:14:09.075722 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:09.075780 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:09.075909 (+ 129us) server_negotiation.cc:299] Negotiation successful
0620 19:14:09.075960 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 19:14:09.076184 30588 leader_election.cc:288] T 00000000000000000000000000000000 P 6b85179a4d754da3945a6658820dbf66 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:14:09.076433 30598 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b85179a4d754da3945a6658820dbf66 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:14:09.076612 30598 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b85179a4d754da3945a6658820dbf66 [term 1 LEADER]: Becoming Leader. State: Replica: 6b85179a4d754da3945a6658820dbf66, State: Running, Role: LEADER
I20260620 19:14:09.076979 30598 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b85179a4d754da3945a6658820dbf66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b85179a4d754da3945a6658820dbf66" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40617 } }
I20260620 19:14:09.077562 30588 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b85179a4d754da3945a6658820dbf66 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:14:09.078153 30598 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b85179a4d754da3945a6658820dbf66 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b85179a4d754da3945a6658820dbf66. Latest consensus state: current_term: 1 leader_uuid: "6b85179a4d754da3945a6658820dbf66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b85179a4d754da3945a6658820dbf66" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40617 } } }
I20260620 19:14:09.078364 30598 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b85179a4d754da3945a6658820dbf66 [sys.catalog]: This master's current role is: LEADER
I20260620 19:14:09.078627 30600 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b85179a4d754da3945a6658820dbf66 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6b85179a4d754da3945a6658820dbf66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b85179a4d754da3945a6658820dbf66" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40617 } } }
I20260620 19:14:09.078781 30600 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b85179a4d754da3945a6658820dbf66 [sys.catalog]: This master's current role is: LEADER
I20260620 19:14:09.078822 30607 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:14:09.082227 30607 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:14:09.088308 30607 catalog_manager.cc:1347] Generated new cluster ID: fe6770b9b3884645a6b1b8136ba4a94a
I20260620 19:14:09.088397 30607 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:14:09.105257 30607 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:14:09.106599 30607 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:14:09.129510 30607 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6b85179a4d754da3945a6658820dbf66: Generated new TSK 0
I20260620 19:14:09.130259 30607 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:14:09.137133 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 30520
2026-06-20T19:14:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1346 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982595231230-9837-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:14:09 dist-test-slave-n3fx krb5kdc[30619](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:14:09 dist-test-slave-n3fx krb5kdc[30619](info): set up 2 sockets
Jun 20 19:14:09 dist-test-slave-n3fx krb5kdc[30619](info): commencing operation
krb5kdc: starting...
W20260620 19:14:10.092453 9837 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982595231230-9837-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982595231230-9837-0/krb5kdc/test-user.keytab.
Jun 20 19:14:10 dist-test-slave-n3fx krb5kdc[30619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982850, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:14:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:14:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982595231230-9837-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982595231230-9837-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 19:14:10.215584 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982595231230-9837-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982595231230-9837-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982595231230-9837-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40263
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982595231230-9837-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40263
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:14:10.328183 30632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:14:10.328529 30632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:14:10.328595 30632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:14:10.332758 30632 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:14:10.332834 30632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:14:10.332860 30632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:14:10.332883 30632 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:14:10.332912 30632 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:14:10.338558 30632 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982595231230-9837-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982595231230-9837-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40263
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982595231230-9837-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982595231230-9837-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:14:10.339716 30632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:14:10.340799 30632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:14:10.349529 30640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:14:10.349732 30632 server_base.cc:1034] running on GCE node
W20260620 19:14:10.349553 30637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:14:10.349532 30638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:14:10.350492 30632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:14:10.351680 30632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:14:10.352931 30632 hybrid_clock.cc:648] HybridClock initialized: now 1781982850352906 us; error 38 us; skew 500 ppm
Jun 20 19:14:10 dist-test-slave-n3fx krb5kdc[30619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982850, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:14:10.356034 30632 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 19:14:10.357054 30632 webserver.cc:460] Webserver started at http://127.0.0.1:43795/ using document root <none> and password file <none>
I20260620 19:14:10.357645 30632 fs_manager.cc:359] Metadata directory not provided
I20260620 19:14:10.357713 30632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:14:10.357923 30632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:14:10.360460 30632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982595231230-9837-0/minicluster-data/master-0/data/instance:
uuid: "310d338333d04d94a3ebe2affd5959c2"
format_stamp: "Formatted at 2026-06-20 19:14:10 on dist-test-slave-n3fx"
server_key: "34c3044bcab05ef493a8e3c59c25325f"
server_key_iv: "e6b8ae4d9f5dc0bf452a4c3e1297d59c"
server_key_version: "encryptionkey@0"
I20260620 19:14:10.361035 30632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982595231230-9837-0/minicluster-data/master-0/wal/instance:
uuid: "310d338333d04d94a3ebe2affd5959c2"
format_stamp: "Formatted at 2026-06-20 19:14:10 on dist-test-slave-n3fx"
server_key: "34c3044bcab05ef493a8e3c59c25325f"
server_key_iv: "e6b8ae4d9f5dc0bf452a4c3e1297d59c"
server_key_version: "encryptionkey@0"
I20260620 19:14:10.365073 30632 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 19:14:10.367870 30647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:10.369345 30632 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 19:14:10.369547 30632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982595231230-9837-0/minicluster-data/master-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982595231230-9837-0/minicluster-data/master-0/wal
uuid: "310d338333d04d94a3ebe2affd5959c2"
format_stamp: "Formatted at 2026-06-20 19:14:10 on dist-test-slave-n3fx"
server_key: "34c3044bcab05ef493a8e3c59c25325f"
server_key_iv: "e6b8ae4d9f5dc0bf452a4c3e1297d59c"
server_key_version: "encryptionkey@0"
I20260620 19:14:10.369674 30632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982595231230-9837-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982595231230-9837-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982595231230-9837-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:14:10.374590 30632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:14:10.377794 30632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:14:10.377978 30632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:14:10.384845 30632 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40263
I20260620 19:14:10.384895 30699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40263 every 8 connection(s)
I20260620 19:14:10.385782 30632 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
I20260620 19:14:10.388607 30700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:14:10.392066 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 30632
I20260620 19:14:10.392187 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982595231230-9837-0/minicluster-data/master-0/wal/instance
I20260620 19:14:10.392477 9837 external_mini_cluster.cc:1428] Setting key 1ee92e61e09a74deb982c9efb60f1875
I20260620 19:14:10.394430 30700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 310d338333d04d94a3ebe2affd5959c2: Bootstrap starting.
I20260620 19:14:10.396924 30700 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 310d338333d04d94a3ebe2affd5959c2: Neither blocks nor log segments found. Creating new log.
I20260620 19:14:10.397660 30700 log.cc:824] T 00000000000000000000000000000000 P 310d338333d04d94a3ebe2affd5959c2: Log is configured to *not* fsync() on all Append() calls
I20260620 19:14:10.397697 30703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:10.393913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53891 (local address 127.0.0.1:40263)
0620 19:14:10.394339 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:10.394353 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:10.394390 (+ 37us) server_negotiation.cc:407] Connection header received
0620 19:14:10.395059 (+ 669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:10.395077 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:10.395375 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:10.395659 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:10.396519 (+ 860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:10.396524 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:10.396536 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:14:10.396547 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:10.396604 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:14:10.396624 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:10.396707 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:10.396827 (+ 120us) server_negotiation.cc:299] Negotiation successful
0620 19:14:10.396997 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":121,"threads_started":1}
I20260620 19:14:10.399605 30700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 310d338333d04d94a3ebe2affd5959c2: No bootstrap required, opened a new log
I20260620 19:14:10.400324 9837 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:14:10.403975 30700 raft_consensus.cc:348] T 00000000000000000000000000000000 P 310d338333d04d94a3ebe2affd5959c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "310d338333d04d94a3ebe2affd5959c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40263 } }
I20260620 19:14:10.404253 30700 raft_consensus.cc:374] T 00000000000000000000000000000000 P 310d338333d04d94a3ebe2affd5959c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:14:10.404330 30700 raft_consensus.cc:729] T 00000000000000000000000000000000 P 310d338333d04d94a3ebe2affd5959c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 310d338333d04d94a3ebe2affd5959c2, State: Initialized, Role: FOLLOWER
I20260620 19:14:10.404464 30703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:10.402760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35755 (local address 127.0.0.1:40263)
0620 19:14:10.402897 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:10.402901 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:10.403088 (+ 187us) server_negotiation.cc:407] Connection header received
0620 19:14:10.403203 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:10.403206 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:10.403289 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:10.403359 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:10.404049 (+ 690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:10.404052 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:10.404055 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:14:10.404066 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:10.404094 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:14:10.404104 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:10.404176 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:10.404306 (+ 130us) server_negotiation.cc:299] Negotiation successful
0620 19:14:10.404347 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 19:14:10.404834 30700 consensus_queue.cc:260] T 00000000000000000000000000000000 P 310d338333d04d94a3ebe2affd5959c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "310d338333d04d94a3ebe2affd5959c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40263 } }
I20260620 19:14:10.404990 30700 raft_consensus.cc:388] T 00000000000000000000000000000000 P 310d338333d04d94a3ebe2affd5959c2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:14:10.405056 30700 raft_consensus.cc:482] T 00000000000000000000000000000000 P 310d338333d04d94a3ebe2affd5959c2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:14:10.405157 30700 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 310d338333d04d94a3ebe2affd5959c2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:14:10.406116 30700 raft_consensus.cc:504] T 00000000000000000000000000000000 P 310d338333d04d94a3ebe2affd5959c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "310d338333d04d94a3ebe2affd5959c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40263 } }
I20260620 19:14:10.406440 30700 leader_election.cc:302] T 00000000000000000000000000000000 P 310d338333d04d94a3ebe2affd5959c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 310d338333d04d94a3ebe2affd5959c2; no voters:
I20260620 19:14:10.406742 30700 leader_election.cc:288] T 00000000000000000000000000000000 P 310d338333d04d94a3ebe2affd5959c2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:14:10.406826 30710 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 310d338333d04d94a3ebe2affd5959c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:14:10.407032 30710 raft_consensus.cc:686] T 00000000000000000000000000000000 P 310d338333d04d94a3ebe2affd5959c2 [term 1 LEADER]: Becoming Leader. State: Replica: 310d338333d04d94a3ebe2affd5959c2, State: Running, Role: LEADER
I20260620 19:14:10.407399 30710 consensus_queue.cc:237] T 00000000000000000000000000000000 P 310d338333d04d94a3ebe2affd5959c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "310d338333d04d94a3ebe2affd5959c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40263 } }
I20260620 19:14:10.408113 30700 sys_catalog.cc:564] T 00000000000000000000000000000000 P 310d338333d04d94a3ebe2affd5959c2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:14:10.409209 30712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 310d338333d04d94a3ebe2affd5959c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 310d338333d04d94a3ebe2affd5959c2. Latest consensus state: current_term: 1 leader_uuid: "310d338333d04d94a3ebe2affd5959c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "310d338333d04d94a3ebe2affd5959c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40263 } } }
I20260620 19:14:10.409238 30711 sys_catalog.cc:455] T 00000000000000000000000000000000 P 310d338333d04d94a3ebe2affd5959c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "310d338333d04d94a3ebe2affd5959c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "310d338333d04d94a3ebe2affd5959c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40263 } } }
I20260620 19:14:10.409387 30712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 310d338333d04d94a3ebe2affd5959c2 [sys.catalog]: This master's current role is: LEADER
I20260620 19:14:10.409415 30711 sys_catalog.cc:458] T 00000000000000000000000000000000 P 310d338333d04d94a3ebe2affd5959c2 [sys.catalog]: This master's current role is: LEADER
W20260620 19:14:10.412588 30725 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 310d338333d04d94a3ebe2affd5959c2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:14:10.412662 30725 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:14:10.412745 30719 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:14:10.413729 30719 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:14:10.420210 30719 catalog_manager.cc:1347] Generated new cluster ID: 7246cc06f71c431c965ca944668317b3
I20260620 19:14:10.420341 30719 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:14:10.436882 30719 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:14:10.437214 30719 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:14:10.447201 30719 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 310d338333d04d94a3ebe2affd5959c2: Generated new TSK 0
I20260620 19:14:10.447981 30719 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:14:10.454964 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 30632
2026-06-20T19:14:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1318 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982595231230-9837-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:14:10 dist-test-slave-n3fx krb5kdc[30731](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:14:10 dist-test-slave-n3fx krb5kdc[30731](info): set up 2 sockets
Jun 20 19:14:10 dist-test-slave-n3fx krb5kdc[30731](info): commencing operation
krb5kdc: starting...
W20260620 19:14:11.412174 9837 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982595231230-9837-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982595231230-9837-0/krb5kdc/test-user.keytab.
Jun 20 19:14:11 dist-test-slave-n3fx krb5kdc[30731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982851, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:14:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:14:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982595231230-9837-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982595231230-9837-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 19:14:11.533448 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982595231230-9837-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982595231230-9837-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982595231230-9837-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39591
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982595231230-9837-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39591
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:14:11.644294 30744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:14:11.644598 30744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:14:11.644641 30744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:14:11.648823 30744 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:14:11.648916 30744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:14:11.648945 30744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:14:11.648967 30744 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:14:11.648985 30744 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:14:11.654457 30744 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982595231230-9837-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982595231230-9837-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39591
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982595231230-9837-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30744
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982595231230-9837-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:14:11.655563 30744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:14:11.656625 30744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:14:11.665412 30750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:14:11.665424 30749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:14:11.665627 30744 server_base.cc:1034] running on GCE node
W20260620 19:14:11.665388 30752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:14:11.666316 30744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:14:11.667438 30744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:14:11.668640 30744 hybrid_clock.cc:648] HybridClock initialized: now 1781982851668625 us; error 29 us; skew 500 ppm
Jun 20 19:14:11 dist-test-slave-n3fx krb5kdc[30731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982851, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:14:11.671537 30744 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 19:14:11.672587 30744 webserver.cc:460] Webserver started at http://127.0.0.1:36689/ using document root <none> and password file <none>
I20260620 19:14:11.673163 30744 fs_manager.cc:359] Metadata directory not provided
I20260620 19:14:11.673233 30744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:14:11.673425 30744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:14:11.675448 30744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982595231230-9837-0/minicluster-data/master-0/data/instance:
uuid: "9d008b6e86e846d8a98edf526f604af2"
format_stamp: "Formatted at 2026-06-20 19:14:11 on dist-test-slave-n3fx"
server_key: "790e15b2bbd86c92deb9c4bdbcc904b5"
server_key_iv: "44769e0c1b37e04eb9e2a1c8d8134268"
server_key_version: "encryptionkey@0"
I20260620 19:14:11.675994 30744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982595231230-9837-0/minicluster-data/master-0/wal/instance:
uuid: "9d008b6e86e846d8a98edf526f604af2"
format_stamp: "Formatted at 2026-06-20 19:14:11 on dist-test-slave-n3fx"
server_key: "790e15b2bbd86c92deb9c4bdbcc904b5"
server_key_iv: "44769e0c1b37e04eb9e2a1c8d8134268"
server_key_version: "encryptionkey@0"
I20260620 19:14:11.679728 30744 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 19:14:11.682384 30759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:11.683763 30744 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 19:14:11.683907 30744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982595231230-9837-0/minicluster-data/master-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982595231230-9837-0/minicluster-data/master-0/wal
uuid: "9d008b6e86e846d8a98edf526f604af2"
format_stamp: "Formatted at 2026-06-20 19:14:11 on dist-test-slave-n3fx"
server_key: "790e15b2bbd86c92deb9c4bdbcc904b5"
server_key_iv: "44769e0c1b37e04eb9e2a1c8d8134268"
server_key_version: "encryptionkey@0"
I20260620 19:14:11.684024 30744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982595231230-9837-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982595231230-9837-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982595231230-9837-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:14:11.711150 30744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:14:11.714290 30744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:14:11.714526 30744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:14:11.721592 30744 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39591
I20260620 19:14:11.721607 30811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39591 every 8 connection(s)
I20260620 19:14:11.722565 30744 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
I20260620 19:14:11.725382 30812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:14:11.730515 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 30744
I20260620 19:14:11.730652 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982595231230-9837-0/minicluster-data/master-0/wal/instance
I20260620 19:14:11.730944 9837 external_mini_cluster.cc:1428] Setting key 53243f9891f246b8f493ee9796e32e9f
I20260620 19:14:11.731308 30812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d008b6e86e846d8a98edf526f604af2: Bootstrap starting.
I20260620 19:14:11.733814 30812 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d008b6e86e846d8a98edf526f604af2: Neither blocks nor log segments found. Creating new log.
I20260620 19:14:11.734678 30812 log.cc:824] T 00000000000000000000000000000000 P 9d008b6e86e846d8a98edf526f604af2: Log is configured to *not* fsync() on all Append() calls
I20260620 19:14:11.736989 30812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d008b6e86e846d8a98edf526f604af2: No bootstrap required, opened a new log
Jun 20 19:14:11 dist-test-slave-n3fx krb5kdc[30731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982851, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 19:14:11.741475 30812 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d008b6e86e846d8a98edf526f604af2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d008b6e86e846d8a98edf526f604af2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39591 } }
I20260620 19:14:11.741746 30812 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d008b6e86e846d8a98edf526f604af2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:14:11.741822 30812 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d008b6e86e846d8a98edf526f604af2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d008b6e86e846d8a98edf526f604af2, State: Initialized, Role: FOLLOWER
I20260620 19:14:11.742334 30812 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d008b6e86e846d8a98edf526f604af2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d008b6e86e846d8a98edf526f604af2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39591 } }
I20260620 19:14:11.742502 30812 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d008b6e86e846d8a98edf526f604af2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:14:11.742568 30812 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d008b6e86e846d8a98edf526f604af2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:14:11.742664 30812 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d008b6e86e846d8a98edf526f604af2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:14:11.743779 30812 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d008b6e86e846d8a98edf526f604af2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d008b6e86e846d8a98edf526f604af2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39591 } }
I20260620 19:14:11.744175 30812 leader_election.cc:302] T 00000000000000000000000000000000 P 9d008b6e86e846d8a98edf526f604af2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d008b6e86e846d8a98edf526f604af2; no voters:
I20260620 19:14:11.744570 30812 leader_election.cc:288] T 00000000000000000000000000000000 P 9d008b6e86e846d8a98edf526f604af2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:14:11.744694 30817 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d008b6e86e846d8a98edf526f604af2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:14:11.744925 30817 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d008b6e86e846d8a98edf526f604af2 [term 1 LEADER]: Becoming Leader. State: Replica: 9d008b6e86e846d8a98edf526f604af2, State: Running, Role: LEADER
I20260620 19:14:11.745277 30817 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d008b6e86e846d8a98edf526f604af2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d008b6e86e846d8a98edf526f604af2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39591 } }
I20260620 19:14:11.746006 30812 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d008b6e86e846d8a98edf526f604af2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:14:11.746903 30818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d008b6e86e846d8a98edf526f604af2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d008b6e86e846d8a98edf526f604af2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d008b6e86e846d8a98edf526f604af2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39591 } } }
I20260620 19:14:11.747047 30818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d008b6e86e846d8a98edf526f604af2 [sys.catalog]: This master's current role is: LEADER
I20260620 19:14:11.747236 30815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:11.732358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.71:35135 (local address 127.0.0.1:39591)
0620 19:14:11.732813 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:11.732831 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:11.732866 (+ 35us) server_negotiation.cc:407] Connection header received
0620 19:14:11.733511 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:11.733528 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:11.733880 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:11.734193 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:11.735050 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:11.736357 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:11.737522 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:11.738054 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:11.738257 (+ 203us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:11.741192 (+ 2935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:11.741198 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:11.741216 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:14:11.741250 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:11.743228 (+ 1978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:11.743748 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:11.743754 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:11.743760 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:11.743945 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:11.744277 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:11.744280 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:11.744282 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:11.744849 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:11.745789 (+ 940us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:11.745966 (+ 177us) server_negotiation.cc:299] Negotiation successful
0620 19:14:11.746202 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":142,"threads_started":1}
I20260620 19:14:11.747320 30819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d008b6e86e846d8a98edf526f604af2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d008b6e86e846d8a98edf526f604af2. Latest consensus state: current_term: 1 leader_uuid: "9d008b6e86e846d8a98edf526f604af2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d008b6e86e846d8a98edf526f604af2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39591 } } }
I20260620 19:14:11.747411 30819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d008b6e86e846d8a98edf526f604af2 [sys.catalog]: This master's current role is: LEADER
I20260620 19:14:11.747443 30826 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:14:11.750756 30826 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:14:11.750958 9837 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:14:11.757141 30826 catalog_manager.cc:1347] Generated new cluster ID: 24fdb9060b6842d69f8cad22ba0a3152
I20260620 19:14:11.757221 30826 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:14:11.763221 30815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:11.753227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.71:44927 (local address 127.0.0.1:39591)
0620 19:14:11.753352 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:11.753357 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:11.753781 (+ 424us) server_negotiation.cc:407] Connection header received
0620 19:14:11.753908 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:11.753911 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:11.753965 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:11.754052 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:11.754851 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:11.756175 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:11.757294 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:11.757678 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:11.757783 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:11.759676 (+ 1893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:11.759680 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:11.759683 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:14:11.759712 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:11.761436 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:11.761951 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:11.761955 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:11.761957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:11.762055 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:11.762350 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:11.762353 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:11.762354 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:11.762559 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:11.762694 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:11.762922 (+ 228us) server_negotiation.cc:299] Negotiation successful
0620 19:14:11.763035 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 19:14:11.784904 30826 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:14:11.785908 30826 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:14:11.801553 30826 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d008b6e86e846d8a98edf526f604af2: Generated new TSK 0
I20260620 19:14:11.802465 30826 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:14:11.809558 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 30744
W20260620 19:14:11.814224 30729 connection.cc:535] client connection to 127.0.0.1:39591 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39591): Connection reset by peer (error 104)
2026-06-20T19:14:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1354 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982595231230-9837-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:14:11 dist-test-slave-n3fx krb5kdc[30843](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:14:11 dist-test-slave-n3fx krb5kdc[30843](info): set up 2 sockets
Jun 20 19:14:11 dist-test-slave-n3fx krb5kdc[30843](info): commencing operation
krb5kdc: starting...
W20260620 19:14:12.788372 9837 mini_kdc.cc:121] Time spent starting KDC: real 0.960s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982595231230-9837-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982595231230-9837-0/krb5kdc/test-user.keytab.
Jun 20 19:14:12 dist-test-slave-n3fx krb5kdc[30843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:14:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:14:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982595231230-9837-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982595231230-9837-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 19:14:12.912079 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982595231230-9837-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982595231230-9837-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982595231230-9837-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35587
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982595231230-9837-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35587
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:14:13.028514 30856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:14:13.028824 30856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:14:13.028887 30856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:14:13.033087 30856 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:14:13.033174 30856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:14:13.033203 30856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:14:13.033226 30856 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:14:13.033246 30856 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:14:13.038901 30856 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982595231230-9837-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982595231230-9837-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35587
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982595231230-9837-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35587
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982595231230-9837-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:14:13.040038 30856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:14:13.041119 30856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:14:13.049926 30864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:14:13.049983 30856 server_base.cc:1034] running on GCE node
W20260620 19:14:13.049932 30862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:14:13.049932 30861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:14:13.050774 30856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:14:13.051906 30856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:14:13.053089 30856 hybrid_clock.cc:648] HybridClock initialized: now 1781982853053076 us; error 30 us; skew 500 ppm
Jun 20 19:14:13 dist-test-slave-n3fx krb5kdc[30843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982853, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:14:13.055967 30856 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 19:14:13.057102 30856 webserver.cc:460] Webserver started at http://127.0.0.1:40903/ using document root <none> and password file <none>
I20260620 19:14:13.057709 30856 fs_manager.cc:359] Metadata directory not provided
I20260620 19:14:13.057787 30856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:14:13.058023 30856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:14:13.060382 30856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982595231230-9837-0/minicluster-data/master-0/data/instance:
uuid: "4437517c79fe45e8a95dfae2bcdee15b"
format_stamp: "Formatted at 2026-06-20 19:14:13 on dist-test-slave-n3fx"
server_key: "0b5f8fa87095d39804b27115564d115c"
server_key_iv: "576426db866ff061e87b95c6543d796e"
server_key_version: "encryptionkey@0"
I20260620 19:14:13.060988 30856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982595231230-9837-0/minicluster-data/master-0/wal/instance:
uuid: "4437517c79fe45e8a95dfae2bcdee15b"
format_stamp: "Formatted at 2026-06-20 19:14:13 on dist-test-slave-n3fx"
server_key: "0b5f8fa87095d39804b27115564d115c"
server_key_iv: "576426db866ff061e87b95c6543d796e"
server_key_version: "encryptionkey@0"
I20260620 19:14:13.065033 30856 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 19:14:13.067845 30871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:13.069252 30856 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:14:13.069398 30856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982595231230-9837-0/minicluster-data/master-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982595231230-9837-0/minicluster-data/master-0/wal
uuid: "4437517c79fe45e8a95dfae2bcdee15b"
format_stamp: "Formatted at 2026-06-20 19:14:13 on dist-test-slave-n3fx"
server_key: "0b5f8fa87095d39804b27115564d115c"
server_key_iv: "576426db866ff061e87b95c6543d796e"
server_key_version: "encryptionkey@0"
I20260620 19:14:13.069516 30856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982595231230-9837-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982595231230-9837-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982595231230-9837-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:14:13.084337 30856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:14:13.087620 30856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:14:13.087826 30856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:14:13.094925 30856 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35587
I20260620 19:14:13.094954 30923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35587 every 8 connection(s)
I20260620 19:14:13.095894 30856 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
I20260620 19:14:13.098737 30924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:14:13.098764 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 30856
I20260620 19:14:13.098867 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982595231230-9837-0/minicluster-data/master-0/wal/instance
I20260620 19:14:13.099157 9837 external_mini_cluster.cc:1428] Setting key 2175a5825abff9b22e985b3f7c673b76
I20260620 19:14:13.105834 30924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4437517c79fe45e8a95dfae2bcdee15b: Bootstrap starting.
I20260620 19:14:13.108743 30924 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4437517c79fe45e8a95dfae2bcdee15b: Neither blocks nor log segments found. Creating new log.
I20260620 19:14:13.109685 30927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:13.100707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.71:37345 (local address 127.0.0.1:35587)
0620 19:14:13.101129 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:13.101147 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:13.101189 (+ 42us) server_negotiation.cc:407] Connection header received
0620 19:14:13.101942 (+ 753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:13.101967 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:13.102409 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:13.102775 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:13.103188 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:13.104666 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:13.105633 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:13.106320 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:13.106482 (+ 162us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:13.107503 (+ 1021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:13.107510 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:13.107527 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:14:13.108053 (+ 526us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:13.108135 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:14:13.108169 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:13.108342 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:13.108548 (+ 206us) server_negotiation.cc:299] Negotiation successful
0620 19:14:13.108699 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":115,"threads_started":1}
I20260620 19:14:13.109663 30924 log.cc:824] T 00000000000000000000000000000000 P 4437517c79fe45e8a95dfae2bcdee15b: Log is configured to *not* fsync() on all Append() calls
I20260620 19:14:13.112013 30924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4437517c79fe45e8a95dfae2bcdee15b: No bootstrap required, opened a new log
I20260620 19:14:13.113166 9837 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:14:13.116163 30924 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4437517c79fe45e8a95dfae2bcdee15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4437517c79fe45e8a95dfae2bcdee15b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35587 } }
I20260620 19:14:13.116539 30924 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4437517c79fe45e8a95dfae2bcdee15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:14:13.116602 30924 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4437517c79fe45e8a95dfae2bcdee15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4437517c79fe45e8a95dfae2bcdee15b, State: Initialized, Role: FOLLOWER
I20260620 19:14:13.117070 30924 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4437517c79fe45e8a95dfae2bcdee15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4437517c79fe45e8a95dfae2bcdee15b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35587 } }
I20260620 19:14:13.117296 30924 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4437517c79fe45e8a95dfae2bcdee15b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:14:13.117401 30924 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4437517c79fe45e8a95dfae2bcdee15b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:14:13.117553 30924 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4437517c79fe45e8a95dfae2bcdee15b [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:14:13.118775 30924 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4437517c79fe45e8a95dfae2bcdee15b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4437517c79fe45e8a95dfae2bcdee15b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35587 } }
I20260620 19:14:13.119184 30924 leader_election.cc:302] T 00000000000000000000000000000000 P 4437517c79fe45e8a95dfae2bcdee15b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4437517c79fe45e8a95dfae2bcdee15b; no voters:
I20260620 19:14:13.119585 30924 leader_election.cc:288] T 00000000000000000000000000000000 P 4437517c79fe45e8a95dfae2bcdee15b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:14:13.119794 30927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:13.115211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.71:34107 (local address 127.0.0.1:35587)
0620 19:14:13.115336 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:13.115340 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:13.115443 (+ 103us) server_negotiation.cc:407] Connection header received
0620 19:14:13.115540 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:13.115543 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:13.115593 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:13.115671 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:13.115918 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:13.116699 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:13.117869 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:13.118236 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:13.118343 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:13.119213 (+ 870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:13.119217 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:13.119219 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:14:13.119236 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:13.119265 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:14:13.119278 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:13.119392 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:13.119611 (+ 219us) server_negotiation.cc:299] Negotiation successful
0620 19:14:13.119653 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 19:14:13.119870 30934 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4437517c79fe45e8a95dfae2bcdee15b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:14:13.120056 30934 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4437517c79fe45e8a95dfae2bcdee15b [term 1 LEADER]: Becoming Leader. State: Replica: 4437517c79fe45e8a95dfae2bcdee15b, State: Running, Role: LEADER
I20260620 19:14:13.120421 30934 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4437517c79fe45e8a95dfae2bcdee15b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4437517c79fe45e8a95dfae2bcdee15b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35587 } }
I20260620 19:14:13.121093 30924 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4437517c79fe45e8a95dfae2bcdee15b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:14:13.121737 30935 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4437517c79fe45e8a95dfae2bcdee15b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4437517c79fe45e8a95dfae2bcdee15b. Latest consensus state: current_term: 1 leader_uuid: "4437517c79fe45e8a95dfae2bcdee15b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4437517c79fe45e8a95dfae2bcdee15b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35587 } } }
I20260620 19:14:13.121868 30935 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4437517c79fe45e8a95dfae2bcdee15b [sys.catalog]: This master's current role is: LEADER
I20260620 19:14:13.122125 30936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4437517c79fe45e8a95dfae2bcdee15b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4437517c79fe45e8a95dfae2bcdee15b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4437517c79fe45e8a95dfae2bcdee15b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35587 } } }
I20260620 19:14:13.122293 30936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4437517c79fe45e8a95dfae2bcdee15b [sys.catalog]: This master's current role is: LEADER
I20260620 19:14:13.122426 30943 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:14:13.127132 30943 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:14:13.133618 30943 catalog_manager.cc:1347] Generated new cluster ID: 38d2cf12ba514ac3acdc03e8fc436e07
I20260620 19:14:13.133697 30943 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:14:13.146364 30943 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:14:13.147341 30943 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:14:13.156332 30943 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4437517c79fe45e8a95dfae2bcdee15b: Generated new TSK 0
I20260620 19:14:13.157070 30943 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:14:13.165441 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 30856
W20260620 19:14:13.170383 30841 connection.cc:535] client connection to 127.0.0.1:35587 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35587): Connection reset by peer (error 104)
2026-06-20T19:14:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1356 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982595231230-9837-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:14:13 dist-test-slave-n3fx krb5kdc[30955](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:14:13 dist-test-slave-n3fx krb5kdc[30955](info): set up 2 sockets
Jun 20 19:14:13 dist-test-slave-n3fx krb5kdc[30955](info): commencing operation
krb5kdc: starting...
W20260620 19:14:14.124286 9837 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982595231230-9837-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982595231230-9837-0/krb5kdc/test-user.keytab.
Jun 20 19:14:14 dist-test-slave-n3fx krb5kdc[30955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:14:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:14:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982595231230-9837-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982595231230-9837-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 19:14:14.244138 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982595231230-9837-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982595231230-9837-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982595231230-9837-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37445
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982595231230-9837-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37445
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:14:14.354733 30968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:14:14.355010 30968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:14:14.355072 30968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:14:14.359201 30968 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:14:14.359282 30968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:14:14.359308 30968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:14:14.359330 30968 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:14:14.359368 30968 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:14:14.364979 30968 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982595231230-9837-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982595231230-9837-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37445
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982595231230-9837-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37445
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982595231230-9837-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:14:14.366166 30968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:14:14.367197 30968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:14:14.376006 30976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:14:14.376000 30973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:14:14.376017 30974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:14:14.376187 30968 server_base.cc:1034] running on GCE node
I20260620 19:14:14.377023 30968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:14:14.378240 30968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:14:14.379427 30968 hybrid_clock.cc:648] HybridClock initialized: now 1781982854379414 us; error 31 us; skew 500 ppm
Jun 20 19:14:14 dist-test-slave-n3fx krb5kdc[30955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982854, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:14:14.387122 30968 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 19:14:14.388159 30968 webserver.cc:460] Webserver started at http://127.0.0.1:42015/ using document root <none> and password file <none>
I20260620 19:14:14.388760 30968 fs_manager.cc:359] Metadata directory not provided
I20260620 19:14:14.388830 30968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:14:14.389032 30968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:14:14.391214 30968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982595231230-9837-0/minicluster-data/master-0/data/instance:
uuid: "1ef0f646b1dc412f840c75ced4e778f4"
format_stamp: "Formatted at 2026-06-20 19:14:14 on dist-test-slave-n3fx"
server_key: "8b470e67e864d35d16c6cb5493f167b9"
server_key_iv: "6dae7eef1154d3d80cd3aa94656fecbe"
server_key_version: "encryptionkey@0"
I20260620 19:14:14.391788 30968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982595231230-9837-0/minicluster-data/master-0/wal/instance:
uuid: "1ef0f646b1dc412f840c75ced4e778f4"
format_stamp: "Formatted at 2026-06-20 19:14:14 on dist-test-slave-n3fx"
server_key: "8b470e67e864d35d16c6cb5493f167b9"
server_key_iv: "6dae7eef1154d3d80cd3aa94656fecbe"
server_key_version: "encryptionkey@0"
I20260620 19:14:14.396574 30968 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 19:14:14.400382 30983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:14.401904 30968 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 19:14:14.402055 30968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982595231230-9837-0/minicluster-data/master-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982595231230-9837-0/minicluster-data/master-0/wal
uuid: "1ef0f646b1dc412f840c75ced4e778f4"
format_stamp: "Formatted at 2026-06-20 19:14:14 on dist-test-slave-n3fx"
server_key: "8b470e67e864d35d16c6cb5493f167b9"
server_key_iv: "6dae7eef1154d3d80cd3aa94656fecbe"
server_key_version: "encryptionkey@0"
I20260620 19:14:14.402197 30968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982595231230-9837-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982595231230-9837-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982595231230-9837-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:14:14.408142 30968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:14:14.412148 30968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:14:14.412369 30968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:14:14.421351 30968 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37445
I20260620 19:14:14.421657 31035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37445 every 8 connection(s)
I20260620 19:14:14.422566 30968 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
I20260620 19:14:14.426210 31036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:14:14.430737 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 30968
I20260620 19:14:14.430871 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982595231230-9837-0/minicluster-data/master-0/wal/instance
I20260620 19:14:14.431162 9837 external_mini_cluster.cc:1428] Setting key a16d244dc24ef9773cece17eb9db4d93
I20260620 19:14:14.433996 31036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ef0f646b1dc412f840c75ced4e778f4: Bootstrap starting.
I20260620 19:14:14.437100 31036 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ef0f646b1dc412f840c75ced4e778f4: Neither blocks nor log segments found. Creating new log.
I20260620 19:14:14.438079 31036 log.cc:824] T 00000000000000000000000000000000 P 1ef0f646b1dc412f840c75ced4e778f4: Log is configured to *not* fsync() on all Append() calls
I20260620 19:14:14.438305 31039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:14.432586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.71:43939 (local address 127.0.0.1:37445)
0620 19:14:14.433079 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:14.433100 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:14.433148 (+ 48us) server_negotiation.cc:407] Connection header received
0620 19:14:14.434128 (+ 980us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:14.434153 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:14.434541 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:14.434946 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:14.436867 (+ 1921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:14.436873 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:14.436890 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:14:14.436902 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:14.436972 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:14:14.436998 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:14.437107 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:14.437221 (+ 114us) server_negotiation.cc:299] Negotiation successful
0620 19:14:14.437386 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":148,"threads_started":1}
I20260620 19:14:14.440287 31036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ef0f646b1dc412f840c75ced4e778f4: No bootstrap required, opened a new log
I20260620 19:14:14.441218 9837 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:14:14.445137 31039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:14.443374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.71:40423 (local address 127.0.0.1:37445)
0620 19:14:14.443484 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:14.443488 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:14.443681 (+ 193us) server_negotiation.cc:407] Connection header received
0620 19:14:14.443762 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:14.443765 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:14.443805 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:14.443888 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:14.444736 (+ 848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:14.444740 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:14.444743 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:14:14.444757 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:14.444789 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:14:14.444802 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:14.444890 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:14.444968 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 19:14:14.445011 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 19:14:14.444938 31036 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ef0f646b1dc412f840c75ced4e778f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ef0f646b1dc412f840c75ced4e778f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37445 } }
I20260620 19:14:14.445418 31036 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ef0f646b1dc412f840c75ced4e778f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:14:14.445494 31036 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ef0f646b1dc412f840c75ced4e778f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ef0f646b1dc412f840c75ced4e778f4, State: Initialized, Role: FOLLOWER
I20260620 19:14:14.445955 31036 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ef0f646b1dc412f840c75ced4e778f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ef0f646b1dc412f840c75ced4e778f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37445 } }
I20260620 19:14:14.446094 31036 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ef0f646b1dc412f840c75ced4e778f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:14:14.446151 31036 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ef0f646b1dc412f840c75ced4e778f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:14:14.446254 31036 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ef0f646b1dc412f840c75ced4e778f4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:14:14.447202 31036 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ef0f646b1dc412f840c75ced4e778f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ef0f646b1dc412f840c75ced4e778f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37445 } }
I20260620 19:14:14.447561 31036 leader_election.cc:302] T 00000000000000000000000000000000 P 1ef0f646b1dc412f840c75ced4e778f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ef0f646b1dc412f840c75ced4e778f4; no voters:
I20260620 19:14:14.447852 31036 leader_election.cc:288] T 00000000000000000000000000000000 P 1ef0f646b1dc412f840c75ced4e778f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:14:14.448052 31046 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ef0f646b1dc412f840c75ced4e778f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:14:14.448282 31046 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ef0f646b1dc412f840c75ced4e778f4 [term 1 LEADER]: Becoming Leader. State: Replica: 1ef0f646b1dc412f840c75ced4e778f4, State: Running, Role: LEADER
I20260620 19:14:14.448587 31046 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ef0f646b1dc412f840c75ced4e778f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ef0f646b1dc412f840c75ced4e778f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37445 } }
I20260620 19:14:14.449111 31036 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ef0f646b1dc412f840c75ced4e778f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:14:14.450110 31048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ef0f646b1dc412f840c75ced4e778f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ef0f646b1dc412f840c75ced4e778f4. Latest consensus state: current_term: 1 leader_uuid: "1ef0f646b1dc412f840c75ced4e778f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ef0f646b1dc412f840c75ced4e778f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37445 } } }
I20260620 19:14:14.450228 31048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ef0f646b1dc412f840c75ced4e778f4 [sys.catalog]: This master's current role is: LEADER
I20260620 19:14:14.450646 31055 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:14:14.450596 31047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ef0f646b1dc412f840c75ced4e778f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ef0f646b1dc412f840c75ced4e778f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ef0f646b1dc412f840c75ced4e778f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37445 } } }
I20260620 19:14:14.450738 31047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ef0f646b1dc412f840c75ced4e778f4 [sys.catalog]: This master's current role is: LEADER
I20260620 19:14:14.454358 31055 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:14:14.459429 31055 catalog_manager.cc:1347] Generated new cluster ID: 6cadcd264f0948ea8993eb8c28628844
I20260620 19:14:14.459502 31055 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:14:14.475951 31055 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:14:14.476311 31055 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:14:14.485427 31055 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ef0f646b1dc412f840c75ced4e778f4: Generated new TSK 0
I20260620 19:14:14.486143 31055 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:14:14.494745 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 30968
2026-06-20T19:14:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1328 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982595231230-9837-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:14:14 dist-test-slave-n3fx krb5kdc[31067](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:14:14 dist-test-slave-n3fx krb5kdc[31067](info): set up 2 sockets
Jun 20 19:14:14 dist-test-slave-n3fx krb5kdc[31067](info): commencing operation
krb5kdc: starting...
W20260620 19:14:15.456193 9837 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982595231230-9837-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982595231230-9837-0/krb5kdc/test-user.keytab.
Jun 20 19:14:15 dist-test-slave-n3fx krb5kdc[31067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:14:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:14:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982595231230-9837-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982595231230-9837-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 19:14:15.575960 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982595231230-9837-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982595231230-9837-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982595231230-9837-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35671
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982595231230-9837-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35671
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:14:15.687273 31080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:14:15.687601 31080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:14:15.687671 31080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:14:15.691843 31080 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:14:15.691921 31080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:14:15.691951 31080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:14:15.691972 31080 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:14:15.691993 31080 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:14:15.697538 31080 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982595231230-9837-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982595231230-9837-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35671
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982595231230-9837-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982595231230-9837-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:14:15.698690 31080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:14:15.699740 31080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:14:15.708288 31088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:14:15.708267 31085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:14:15.708274 31086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:14:15.708561 31080 server_base.cc:1034] running on GCE node
I20260620 19:14:15.709267 31080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:14:15.710427 31080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:14:15.711617 31080 hybrid_clock.cc:648] HybridClock initialized: now 1781982855711599 us; error 31 us; skew 500 ppm
Jun 20 19:14:15 dist-test-slave-n3fx krb5kdc[31067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982855, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:14:15.714455 31080 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 19:14:15.715400 31080 webserver.cc:460] Webserver started at http://127.0.0.1:33499/ using document root <none> and password file <none>
I20260620 19:14:15.716050 31080 fs_manager.cc:359] Metadata directory not provided
I20260620 19:14:15.716116 31080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:14:15.716320 31080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:14:15.718410 31080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982595231230-9837-0/minicluster-data/master-0/data/instance:
uuid: "7b6859e2d5ea4ba7955ca65c35ae9eb8"
format_stamp: "Formatted at 2026-06-20 19:14:15 on dist-test-slave-n3fx"
server_key: "a64eaab85667a63e29ffa0136fecb577"
server_key_iv: "af3650388b62c641290b49450f956e67"
server_key_version: "encryptionkey@0"
I20260620 19:14:15.718984 31080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982595231230-9837-0/minicluster-data/master-0/wal/instance:
uuid: "7b6859e2d5ea4ba7955ca65c35ae9eb8"
format_stamp: "Formatted at 2026-06-20 19:14:15 on dist-test-slave-n3fx"
server_key: "a64eaab85667a63e29ffa0136fecb577"
server_key_iv: "af3650388b62c641290b49450f956e67"
server_key_version: "encryptionkey@0"
I20260620 19:14:15.722901 31080 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 19:14:15.725514 31095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:15.726881 31080 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 19:14:15.727036 31080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982595231230-9837-0/minicluster-data/master-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982595231230-9837-0/minicluster-data/master-0/wal
uuid: "7b6859e2d5ea4ba7955ca65c35ae9eb8"
format_stamp: "Formatted at 2026-06-20 19:14:15 on dist-test-slave-n3fx"
server_key: "a64eaab85667a63e29ffa0136fecb577"
server_key_iv: "af3650388b62c641290b49450f956e67"
server_key_version: "encryptionkey@0"
I20260620 19:14:15.727159 31080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982595231230-9837-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982595231230-9837-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982595231230-9837-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:14:15.739718 31080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:14:15.742800 31080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:14:15.742980 31080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:14:15.749502 31080 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35671
I20260620 19:14:15.749567 31147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35671 every 8 connection(s)
I20260620 19:14:15.750607 31080 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
I20260620 19:14:15.752472 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 31080
I20260620 19:14:15.752594 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982595231230-9837-0/minicluster-data/master-0/wal/instance
I20260620 19:14:15.752926 9837 external_mini_cluster.cc:1428] Setting key 8c6480927c4d8c1403d58a3945c69f5d
I20260620 19:14:15.754123 31148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:14:15.761029 31148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b6859e2d5ea4ba7955ca65c35ae9eb8: Bootstrap starting.
Jun 20 19:14:15 dist-test-slave-n3fx krb5kdc[31067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 19:14:15.763468 31148 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b6859e2d5ea4ba7955ca65c35ae9eb8: Neither blocks nor log segments found. Creating new log.
I20260620 19:14:15.764443 31148 log.cc:824] T 00000000000000000000000000000000 P 7b6859e2d5ea4ba7955ca65c35ae9eb8: Log is configured to *not* fsync() on all Append() calls
I20260620 19:14:15.766803 31148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b6859e2d5ea4ba7955ca65c35ae9eb8: No bootstrap required, opened a new log
I20260620 19:14:15.768388 31151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:15.754341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.71:47609 (local address 127.0.0.1:35671)
0620 19:14:15.754809 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:15.754828 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:15.754869 (+ 41us) server_negotiation.cc:407] Connection header received
0620 19:14:15.755721 (+ 852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:15.755746 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:15.756204 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:15.756623 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:15.757319 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:15.758825 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:15.759720 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:15.760445 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:15.760599 (+ 154us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:15.763203 (+ 2604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:15.763210 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:15.763227 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:14:15.763264 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:15.765191 (+ 1927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:15.765686 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:15.765692 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:15.765698 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:15.765843 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:15.766149 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:15.766153 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:15.766154 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:15.766660 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:15.766858 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:15.767140 (+ 282us) server_negotiation.cc:299] Negotiation successful
0620 19:14:15.767411 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":128,"threads_started":1}
I20260620 19:14:15.771230 31148 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b6859e2d5ea4ba7955ca65c35ae9eb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b6859e2d5ea4ba7955ca65c35ae9eb8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35671 } }
I20260620 19:14:15.771502 31148 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b6859e2d5ea4ba7955ca65c35ae9eb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:14:15.771560 31148 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b6859e2d5ea4ba7955ca65c35ae9eb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b6859e2d5ea4ba7955ca65c35ae9eb8, State: Initialized, Role: FOLLOWER
I20260620 19:14:15.771871 9837 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:14:15.772029 31148 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b6859e2d5ea4ba7955ca65c35ae9eb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b6859e2d5ea4ba7955ca65c35ae9eb8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35671 } }
I20260620 19:14:15.772176 31148 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b6859e2d5ea4ba7955ca65c35ae9eb8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:14:15.772262 31148 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b6859e2d5ea4ba7955ca65c35ae9eb8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:14:15.772367 31148 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b6859e2d5ea4ba7955ca65c35ae9eb8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:14:15.773392 31148 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b6859e2d5ea4ba7955ca65c35ae9eb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b6859e2d5ea4ba7955ca65c35ae9eb8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35671 } }
I20260620 19:14:15.773846 31148 leader_election.cc:302] T 00000000000000000000000000000000 P 7b6859e2d5ea4ba7955ca65c35ae9eb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b6859e2d5ea4ba7955ca65c35ae9eb8; no voters:
I20260620 19:14:15.774211 31148 leader_election.cc:288] T 00000000000000000000000000000000 P 7b6859e2d5ea4ba7955ca65c35ae9eb8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:14:15.774305 31158 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b6859e2d5ea4ba7955ca65c35ae9eb8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:14:15.774526 31158 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b6859e2d5ea4ba7955ca65c35ae9eb8 [term 1 LEADER]: Becoming Leader. State: Replica: 7b6859e2d5ea4ba7955ca65c35ae9eb8, State: Running, Role: LEADER
I20260620 19:14:15.774884 31158 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b6859e2d5ea4ba7955ca65c35ae9eb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b6859e2d5ea4ba7955ca65c35ae9eb8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35671 } }
I20260620 19:14:15.775698 31148 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b6859e2d5ea4ba7955ca65c35ae9eb8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:14:15.776562 31160 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b6859e2d5ea4ba7955ca65c35ae9eb8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b6859e2d5ea4ba7955ca65c35ae9eb8. Latest consensus state: current_term: 1 leader_uuid: "7b6859e2d5ea4ba7955ca65c35ae9eb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b6859e2d5ea4ba7955ca65c35ae9eb8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35671 } } }
I20260620 19:14:15.776540 31159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b6859e2d5ea4ba7955ca65c35ae9eb8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b6859e2d5ea4ba7955ca65c35ae9eb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b6859e2d5ea4ba7955ca65c35ae9eb8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35671 } } }
I20260620 19:14:15.776716 31160 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b6859e2d5ea4ba7955ca65c35ae9eb8 [sys.catalog]: This master's current role is: LEADER
I20260620 19:14:15.776778 31159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b6859e2d5ea4ba7955ca65c35ae9eb8 [sys.catalog]: This master's current role is: LEADER
I20260620 19:14:15.777137 31167 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:14:15.780807 31167 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:14:15.784245 31151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:15.773971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.71:47669 (local address 127.0.0.1:35671)
0620 19:14:15.774091 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:15.774096 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:15.774254 (+ 158us) server_negotiation.cc:407] Connection header received
0620 19:14:15.774567 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:15.774570 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:15.774622 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:15.774699 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:15.775624 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:15.776426 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:15.777673 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:15.778021 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:15.778098 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:15.780394 (+ 2296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:15.780398 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:15.780401 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:14:15.780429 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:15.782105 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:15.782567 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:15.782570 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:15.782572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:15.782649 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:15.782940 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:15.782944 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:15.782945 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:15.783146 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:15.783275 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:15.783971 (+ 696us) server_negotiation.cc:299] Negotiation successful
0620 19:14:15.784065 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 19:14:15.787425 31167 catalog_manager.cc:1347] Generated new cluster ID: 2779b5c4b30042b8b66825f4ceb0b35c
I20260620 19:14:15.787490 31167 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:14:15.823222 31167 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:14:15.824299 31167 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:14:15.832003 31167 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b6859e2d5ea4ba7955ca65c35ae9eb8: Generated new TSK 0
I20260620 19:14:15.832767 31167 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:14:15.837750 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 31080
W20260620 19:14:15.842388 31065 connection.cc:535] client connection to 127.0.0.1:35671 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35671): Connection reset by peer (error 104)
2026-06-20T19:14:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1343 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982595231230-9837-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:14:15 dist-test-slave-n3fx krb5kdc[31179](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:14:15 dist-test-slave-n3fx krb5kdc[31179](info): set up 2 sockets
Jun 20 19:14:15 dist-test-slave-n3fx krb5kdc[31179](info): commencing operation
krb5kdc: starting...
W20260620 19:14:16.792315 9837 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982595231230-9837-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982595231230-9837-0/krb5kdc/test-user.keytab.
Jun 20 19:14:16 dist-test-slave-n3fx krb5kdc[31179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:14:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:14:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982595231230-9837-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982595231230-9837-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 19:14:16.909305 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982595231230-9837-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982595231230-9837-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982595231230-9837-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39683
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982595231230-9837-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39683
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:14:17.018115 31192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:14:17.018424 31192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:14:17.018467 31192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:14:17.022627 31192 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:14:17.022715 31192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:14:17.022734 31192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:14:17.022753 31192 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:14:17.022768 31192 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:14:17.028370 31192 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982595231230-9837-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982595231230-9837-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39683
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982595231230-9837-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982595231230-9837-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:14:17.029492 31192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:14:17.030499 31192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:14:17.039348 31200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:14:17.039393 31192 server_base.cc:1034] running on GCE node
W20260620 19:14:17.039397 31198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:14:17.039678 31197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:14:17.040302 31192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:14:17.041442 31192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:14:17.042629 31192 hybrid_clock.cc:648] HybridClock initialized: now 1781982857042614 us; error 33 us; skew 500 ppm
Jun 20 19:14:17 dist-test-slave-n3fx krb5kdc[31179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982857, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:14:17.045589 31192 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 19:14:17.046525 31192 webserver.cc:460] Webserver started at http://127.0.0.1:42183/ using document root <none> and password file <none>
I20260620 19:14:17.047048 31192 fs_manager.cc:359] Metadata directory not provided
I20260620 19:14:17.047114 31192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:14:17.047309 31192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:14:17.049453 31192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982595231230-9837-0/minicluster-data/master-0/data/instance:
uuid: "9153f6898d0c412d8634b053f69c447c"
format_stamp: "Formatted at 2026-06-20 19:14:17 on dist-test-slave-n3fx"
server_key: "5513205d395ed64511ee90ebd7f28533"
server_key_iv: "2eb5c46f010fcbbc8ff66851a04f847a"
server_key_version: "encryptionkey@0"
I20260620 19:14:17.050000 31192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982595231230-9837-0/minicluster-data/master-0/wal/instance:
uuid: "9153f6898d0c412d8634b053f69c447c"
format_stamp: "Formatted at 2026-06-20 19:14:17 on dist-test-slave-n3fx"
server_key: "5513205d395ed64511ee90ebd7f28533"
server_key_iv: "2eb5c46f010fcbbc8ff66851a04f847a"
server_key_version: "encryptionkey@0"
I20260620 19:14:17.053876 31192 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 19:14:17.056502 31207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:17.057894 31192 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:14:17.058024 31192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982595231230-9837-0/minicluster-data/master-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982595231230-9837-0/minicluster-data/master-0/wal
uuid: "9153f6898d0c412d8634b053f69c447c"
format_stamp: "Formatted at 2026-06-20 19:14:17 on dist-test-slave-n3fx"
server_key: "5513205d395ed64511ee90ebd7f28533"
server_key_iv: "2eb5c46f010fcbbc8ff66851a04f847a"
server_key_version: "encryptionkey@0"
I20260620 19:14:17.058143 31192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982595231230-9837-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982595231230-9837-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982595231230-9837-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:14:17.085175 31192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:14:17.088658 31192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:14:17.088858 31192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:14:17.095522 31192 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39683
I20260620 19:14:17.095569 31259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39683 every 8 connection(s)
I20260620 19:14:17.096524 31192 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
I20260620 19:14:17.099258 31260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:14:17.104863 31260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9153f6898d0c412d8634b053f69c447c: Bootstrap starting.
I20260620 19:14:17.105747 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 31192
I20260620 19:14:17.105881 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982595231230-9837-0/minicluster-data/master-0/wal/instance
I20260620 19:14:17.106202 9837 external_mini_cluster.cc:1428] Setting key 7f390a771374fc6f3bc4bac1fdd8af19
I20260620 19:14:17.107831 31260 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9153f6898d0c412d8634b053f69c447c: Neither blocks nor log segments found. Creating new log.
I20260620 19:14:17.108600 31260 log.cc:824] T 00000000000000000000000000000000 P 9153f6898d0c412d8634b053f69c447c: Log is configured to *not* fsync() on all Append() calls
I20260620 19:14:17.110761 31260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9153f6898d0c412d8634b053f69c447c: No bootstrap required, opened a new log
I20260620 19:14:17.115068 31260 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9153f6898d0c412d8634b053f69c447c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9153f6898d0c412d8634b053f69c447c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39683 } }
I20260620 19:14:17.115366 31260 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9153f6898d0c412d8634b053f69c447c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:14:17.115438 31260 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9153f6898d0c412d8634b053f69c447c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9153f6898d0c412d8634b053f69c447c, State: Initialized, Role: FOLLOWER
I20260620 19:14:17.115932 31260 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9153f6898d0c412d8634b053f69c447c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9153f6898d0c412d8634b053f69c447c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39683 } }
I20260620 19:14:17.116050 31263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:17.107417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.71:60517 (local address 127.0.0.1:39683)
0620 19:14:17.107841 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:17.107856 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:17.107896 (+ 40us) server_negotiation.cc:407] Connection header received
0620 19:14:17.108628 (+ 732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:17.108648 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:17.109024 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:17.109330 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:17.109755 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:17.111142 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:17.112111 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:17.112862 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:17.113028 (+ 166us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:17.113878 (+ 850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:17.113885 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:17.113903 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:14:17.114459 (+ 556us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:17.114536 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:14:17.114567 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:17.114703 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:17.114888 (+ 185us) server_negotiation.cc:299] Negotiation successful
0620 19:14:17.115024 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":123,"threads_started":1}
I20260620 19:14:17.116111 31260 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9153f6898d0c412d8634b053f69c447c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:14:17.116170 31260 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9153f6898d0c412d8634b053f69c447c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:14:17.116371 31260 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9153f6898d0c412d8634b053f69c447c [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:14:17.117826 31260 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9153f6898d0c412d8634b053f69c447c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9153f6898d0c412d8634b053f69c447c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39683 } }
I20260620 19:14:17.118325 31260 leader_election.cc:302] T 00000000000000000000000000000000 P 9153f6898d0c412d8634b053f69c447c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9153f6898d0c412d8634b053f69c447c; no voters:
I20260620 19:14:17.118778 31260 leader_election.cc:288] T 00000000000000000000000000000000 P 9153f6898d0c412d8634b053f69c447c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:14:17.118886 31265 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9153f6898d0c412d8634b053f69c447c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:14:17.119102 31265 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9153f6898d0c412d8634b053f69c447c [term 1 LEADER]: Becoming Leader. State: Replica: 9153f6898d0c412d8634b053f69c447c, State: Running, Role: LEADER
I20260620 19:14:17.119424 31265 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9153f6898d0c412d8634b053f69c447c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9153f6898d0c412d8634b053f69c447c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39683 } }
I20260620 19:14:17.119951 9837 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:14:17.120146 31260 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9153f6898d0c412d8634b053f69c447c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:14:17.121035 31267 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9153f6898d0c412d8634b053f69c447c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9153f6898d0c412d8634b053f69c447c. Latest consensus state: current_term: 1 leader_uuid: "9153f6898d0c412d8634b053f69c447c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9153f6898d0c412d8634b053f69c447c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39683 } } }
I20260620 19:14:17.121152 31267 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9153f6898d0c412d8634b053f69c447c [sys.catalog]: This master's current role is: LEADER
I20260620 19:14:17.121476 31266 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9153f6898d0c412d8634b053f69c447c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9153f6898d0c412d8634b053f69c447c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9153f6898d0c412d8634b053f69c447c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39683 } } }
I20260620 19:14:17.121557 31266 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9153f6898d0c412d8634b053f69c447c [sys.catalog]: This master's current role is: LEADER
I20260620 19:14:17.121832 31277 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:14:17.124954 31277 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:14:17.127919 31263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:17.122460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.71:39141 (local address 127.0.0.1:39683)
0620 19:14:17.122574 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:17.122579 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:17.123543 (+ 964us) server_negotiation.cc:407] Connection header received
0620 19:14:17.123724 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:17.123727 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:17.123781 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:17.123862 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:17.124272 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:17.124813 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:17.125975 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:17.126277 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:17.126368 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:17.127439 (+ 1071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:17.127442 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:17.127445 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:14:17.127461 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:17.127489 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:14:17.127503 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:17.127595 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:17.127737 (+ 142us) server_negotiation.cc:299] Negotiation successful
0620 19:14:17.127773 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 19:14:17.131335 31277 catalog_manager.cc:1347] Generated new cluster ID: f889e760357c412cb1533419e3feb5f0
I20260620 19:14:17.131408 31277 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:14:17.153913 31277 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:14:17.154874 31277 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:14:17.163463 31277 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9153f6898d0c412d8634b053f69c447c: Generated new TSK 0
I20260620 19:14:17.164417 31277 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:14:17.175365 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 31192
W20260620 19:14:17.179960 31177 connection.cc:535] client connection to 127.0.0.1:39683 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39683): Connection reset by peer (error 104)
2026-06-20T19:14:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1337 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982595231230-9837-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:14:17 dist-test-slave-n3fx krb5kdc[31291](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:14:17 dist-test-slave-n3fx krb5kdc[31291](info): set up 2 sockets
Jun 20 19:14:17 dist-test-slave-n3fx krb5kdc[31291](info): commencing operation
krb5kdc: starting...
W20260620 19:14:18.128348 9837 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982595231230-9837-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982595231230-9837-0/krb5kdc/test-user.keytab.
Jun 20 19:14:18 dist-test-slave-n3fx krb5kdc[31291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982858, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:14:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:14:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982595231230-9837-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982595231230-9837-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 19:14:18.250969 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982595231230-9837-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982595231230-9837-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982595231230-9837-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42479
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982595231230-9837-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42479
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:14:18.369294 31304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:14:18.369598 31304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:14:18.369660 31304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:14:18.373950 31304 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:14:18.374037 31304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:14:18.374056 31304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:14:18.374073 31304 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:14:18.374086 31304 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:14:18.379689 31304 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982595231230-9837-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982595231230-9837-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42479
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982595231230-9837-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31304
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982595231230-9837-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:14:18.380815 31304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:14:18.381788 31304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:14:18.390551 31310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:14:18.390694 31304 server_base.cc:1034] running on GCE node
W20260620 19:14:18.390558 31309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:14:18.390909 31312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:14:18.391556 31304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:14:18.392802 31304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:14:18.393980 31304 hybrid_clock.cc:648] HybridClock initialized: now 1781982858393955 us; error 40 us; skew 500 ppm
Jun 20 19:14:18 dist-test-slave-n3fx krb5kdc[31291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982858, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:14:18.396930 31304 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 19:14:18.397879 31304 webserver.cc:460] Webserver started at http://127.0.0.1:33961/ using document root <none> and password file <none>
I20260620 19:14:18.398409 31304 fs_manager.cc:359] Metadata directory not provided
I20260620 19:14:18.398468 31304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:14:18.398644 31304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:14:18.400755 31304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982595231230-9837-0/minicluster-data/master-0/data/instance:
uuid: "7a603c827f5f4ddc88d2fce82b927e9b"
format_stamp: "Formatted at 2026-06-20 19:14:18 on dist-test-slave-n3fx"
server_key: "f0ae20f6d4c9fc993e83e541777be2a0"
server_key_iv: "d862d2ffda0b80243c72f546bd695f12"
server_key_version: "encryptionkey@0"
I20260620 19:14:18.401283 31304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982595231230-9837-0/minicluster-data/master-0/wal/instance:
uuid: "7a603c827f5f4ddc88d2fce82b927e9b"
format_stamp: "Formatted at 2026-06-20 19:14:18 on dist-test-slave-n3fx"
server_key: "f0ae20f6d4c9fc993e83e541777be2a0"
server_key_iv: "d862d2ffda0b80243c72f546bd695f12"
server_key_version: "encryptionkey@0"
I20260620 19:14:18.405262 31304 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 19:14:18.408036 31319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:18.409504 31304 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 19:14:18.409701 31304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982595231230-9837-0/minicluster-data/master-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982595231230-9837-0/minicluster-data/master-0/wal
uuid: "7a603c827f5f4ddc88d2fce82b927e9b"
format_stamp: "Formatted at 2026-06-20 19:14:18 on dist-test-slave-n3fx"
server_key: "f0ae20f6d4c9fc993e83e541777be2a0"
server_key_iv: "d862d2ffda0b80243c72f546bd695f12"
server_key_version: "encryptionkey@0"
I20260620 19:14:18.409828 31304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982595231230-9837-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982595231230-9837-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982595231230-9837-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:14:18.414685 31304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:14:18.418001 31304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:14:18.418190 31304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:14:18.425299 31304 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42479
I20260620 19:14:18.425349 31371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42479 every 8 connection(s)
I20260620 19:14:18.426246 31304 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
I20260620 19:14:18.427353 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 31304
I20260620 19:14:18.427500 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982595231230-9837-0/minicluster-data/master-0/wal/instance
I20260620 19:14:18.427825 9837 external_mini_cluster.cc:1428] Setting key da840adcfee3d6b314a9cf6b5d51c88a
I20260620 19:14:18.429610 31372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:14:18.433667 31375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:18.429301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.71:45553 (local address 127.0.0.1:42479)
0620 19:14:18.429728 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:18.429747 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:18.429867 (+ 120us) server_negotiation.cc:407] Connection header received
0620 19:14:18.430746 (+ 879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:18.430768 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:18.431138 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:18.431511 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:18.432198 (+ 687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:18.432204 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:18.432242 (+ 38us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:14:18.432253 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:18.432311 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:14:18.432336 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:18.432484 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:18.432552 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 19:14:18.432716 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":110,"threads_started":1}
I20260620 19:14:18.436408 31372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a603c827f5f4ddc88d2fce82b927e9b: Bootstrap starting.
I20260620 19:14:18.436725 9837 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:14:18.438824 31372 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a603c827f5f4ddc88d2fce82b927e9b: Neither blocks nor log segments found. Creating new log.
I20260620 19:14:18.439738 31372 log.cc:824] T 00000000000000000000000000000000 P 7a603c827f5f4ddc88d2fce82b927e9b: Log is configured to *not* fsync() on all Append() calls
I20260620 19:14:18.440395 31375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:18.438724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.71:38015 (local address 127.0.0.1:42479)
0620 19:14:18.438861 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:18.438866 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:18.438996 (+ 130us) server_negotiation.cc:407] Connection header received
0620 19:14:18.439106 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:18.439109 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:18.439153 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:18.439240 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:18.439978 (+ 738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:18.439982 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:18.439985 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:14:18.439997 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:18.440028 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:14:18.440040 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:18.440147 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:18.440183 (+ 36us) server_negotiation.cc:299] Negotiation successful
0620 19:14:18.440256 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 19:14:18.442126 31372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a603c827f5f4ddc88d2fce82b927e9b: No bootstrap required, opened a new log
I20260620 19:14:18.446041 31372 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a603c827f5f4ddc88d2fce82b927e9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a603c827f5f4ddc88d2fce82b927e9b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42479 } }
I20260620 19:14:18.446295 31372 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a603c827f5f4ddc88d2fce82b927e9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:14:18.446329 31372 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a603c827f5f4ddc88d2fce82b927e9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a603c827f5f4ddc88d2fce82b927e9b, State: Initialized, Role: FOLLOWER
I20260620 19:14:18.446705 31372 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a603c827f5f4ddc88d2fce82b927e9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a603c827f5f4ddc88d2fce82b927e9b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42479 } }
I20260620 19:14:18.446821 31372 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a603c827f5f4ddc88d2fce82b927e9b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:14:18.446861 31372 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a603c827f5f4ddc88d2fce82b927e9b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:14:18.446939 31372 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a603c827f5f4ddc88d2fce82b927e9b [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:14:18.447870 31372 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a603c827f5f4ddc88d2fce82b927e9b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a603c827f5f4ddc88d2fce82b927e9b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42479 } }
I20260620 19:14:18.448194 31372 leader_election.cc:302] T 00000000000000000000000000000000 P 7a603c827f5f4ddc88d2fce82b927e9b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a603c827f5f4ddc88d2fce82b927e9b; no voters:
I20260620 19:14:18.448563 31372 leader_election.cc:288] T 00000000000000000000000000000000 P 7a603c827f5f4ddc88d2fce82b927e9b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:14:18.448755 31382 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a603c827f5f4ddc88d2fce82b927e9b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:14:18.448971 31382 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a603c827f5f4ddc88d2fce82b927e9b [term 1 LEADER]: Becoming Leader. State: Replica: 7a603c827f5f4ddc88d2fce82b927e9b, State: Running, Role: LEADER
I20260620 19:14:18.449275 31382 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a603c827f5f4ddc88d2fce82b927e9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a603c827f5f4ddc88d2fce82b927e9b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42479 } }
I20260620 19:14:18.449819 31372 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a603c827f5f4ddc88d2fce82b927e9b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:14:18.450913 31383 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a603c827f5f4ddc88d2fce82b927e9b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a603c827f5f4ddc88d2fce82b927e9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a603c827f5f4ddc88d2fce82b927e9b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42479 } } }
I20260620 19:14:18.451100 31383 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a603c827f5f4ddc88d2fce82b927e9b [sys.catalog]: This master's current role is: LEADER
I20260620 19:14:18.450914 31384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a603c827f5f4ddc88d2fce82b927e9b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a603c827f5f4ddc88d2fce82b927e9b. Latest consensus state: current_term: 1 leader_uuid: "7a603c827f5f4ddc88d2fce82b927e9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a603c827f5f4ddc88d2fce82b927e9b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42479 } } }
I20260620 19:14:18.451445 31384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a603c827f5f4ddc88d2fce82b927e9b [sys.catalog]: This master's current role is: LEADER
W20260620 19:14:18.454526 31397 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7a603c827f5f4ddc88d2fce82b927e9b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:14:18.454627 31397 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:14:18.454716 31391 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:14:18.455444 31391 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:14:18.461161 31391 catalog_manager.cc:1347] Generated new cluster ID: 5cc27f1fdb5e478191bf8cdcd4926c84
I20260620 19:14:18.461251 31391 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:14:18.478380 31391 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:14:18.478744 31391 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:14:18.492141 31391 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a603c827f5f4ddc88d2fce82b927e9b: Generated new TSK 0
I20260620 19:14:18.493108 31391 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:14:18.498420 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 31304
2026-06-20T19:14:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1323 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982595231230-9837-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:14:18 dist-test-slave-n3fx krb5kdc[31403](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:14:18 dist-test-slave-n3fx krb5kdc[31403](info): set up 2 sockets
Jun 20 19:14:18 dist-test-slave-n3fx krb5kdc[31403](info): commencing operation
krb5kdc: starting...
W20260620 19:14:19.460165 9837 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982595231230-9837-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982595231230-9837-0/krb5kdc/test-user.keytab.
Jun 20 19:14:19 dist-test-slave-n3fx krb5kdc[31403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982859, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:14:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:14:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.155.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.155.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.155.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab.
Entry for principal kudu/127.9.155.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab.
I20260620 19:14:19.580843 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982595231230-9837-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982595231230-9837-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982595231230-9837-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:42083
--webserver_interface=127.9.155.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab
--principal=kudu/127.9.155.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:41633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.155.126:42083
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:14:19.691922 31416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:14:19.692251 31416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:14:19.692317 31416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:14:19.696499 31416 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:14:19.696570 31416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:14:19.696590 31416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:14:19.696606 31416 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:14:19.696621 31416 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:14:19.702147 31416 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:41633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982595231230-9837-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982595231230-9837-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.155.126:42083
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab
--principal=kudu/127.9.155.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:42083
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.155.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31416
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982595231230-9837-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:14:19.703173 31416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:14:19.704107 31416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:14:19.712405 31424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:14:19.712438 31421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:14:19.712493 31416 server_base.cc:1034] running on GCE node
W20260620 19:14:19.712438 31422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:14:19.713234 31416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:14:19.714298 31416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:14:19.715453 31416 hybrid_clock.cc:648] HybridClock initialized: now 1781982859715428 us; error 37 us; skew 500 ppm
Jun 20 19:14:19 dist-test-slave-n3fx krb5kdc[31403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982859, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:14:19.722888 31416 init.cc:377] Logged in from keytab as kudu/127.9.155.126@KRBTEST.COM (short username kudu)
I20260620 19:14:19.723905 31416 webserver.cc:460] Webserver started at http://127.9.155.126:40561/ using document root <none> and password file <none>
I20260620 19:14:19.724498 31416 fs_manager.cc:359] Metadata directory not provided
I20260620 19:14:19.724565 31416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:14:19.724766 31416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:14:19.726905 31416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982595231230-9837-0/minicluster-data/master-0/data/instance:
uuid: "4aac4b65141a4eed9496f4343bd9a39f"
format_stamp: "Formatted at 2026-06-20 19:14:19 on dist-test-slave-n3fx"
server_key: "9efcce3c769cba2b6dbd6e4d16805590"
server_key_iv: "2922ed0385ae3868aca8290f295fcbec"
server_key_version: "encryptionkey@0"
I20260620 19:14:19.727478 31416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982595231230-9837-0/minicluster-data/master-0/wal/instance:
uuid: "4aac4b65141a4eed9496f4343bd9a39f"
format_stamp: "Formatted at 2026-06-20 19:14:19 on dist-test-slave-n3fx"
server_key: "9efcce3c769cba2b6dbd6e4d16805590"
server_key_iv: "2922ed0385ae3868aca8290f295fcbec"
server_key_version: "encryptionkey@0"
I20260620 19:14:19.731376 31416 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 19:14:19.733767 31431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:19.734958 31416 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 19:14:19.735088 31416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982595231230-9837-0/minicluster-data/master-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982595231230-9837-0/minicluster-data/master-0/wal
uuid: "4aac4b65141a4eed9496f4343bd9a39f"
format_stamp: "Formatted at 2026-06-20 19:14:19 on dist-test-slave-n3fx"
server_key: "9efcce3c769cba2b6dbd6e4d16805590"
server_key_iv: "2922ed0385ae3868aca8290f295fcbec"
server_key_version: "encryptionkey@0"
I20260620 19:14:19.735186 31416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982595231230-9837-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982595231230-9837-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982595231230-9837-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:14:19.747419 31416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:14:19.750295 31416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:14:19.750468 31416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:14:19.757342 31416 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.126:42083
I20260620 19:14:19.757368 31483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.126:42083 every 8 connection(s)
I20260620 19:14:19.758340 31416 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
I20260620 19:14:19.761059 31484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:14:19.767318 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 31416
I20260620 19:14:19.767467 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982595231230-9837-0/minicluster-data/master-0/wal/instance
I20260620 19:14:19.767602 31484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4aac4b65141a4eed9496f4343bd9a39f: Bootstrap starting.
I20260620 19:14:19.767771 9837 external_mini_cluster.cc:1428] Setting key b4d6e4165cb69001479744673caa7fba
I20260620 19:14:19.770447 31484 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4aac4b65141a4eed9496f4343bd9a39f: Neither blocks nor log segments found. Creating new log.
I20260620 19:14:19.771339 31484 log.cc:824] T 00000000000000000000000000000000 P 4aac4b65141a4eed9496f4343bd9a39f: Log is configured to *not* fsync() on all Append() calls
I20260620 19:14:19.773731 31484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4aac4b65141a4eed9496f4343bd9a39f: No bootstrap required, opened a new log
Jun 20 19:14:19 dist-test-slave-n3fx krb5kdc[31403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982859, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:14:19.778263 31484 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4aac4b65141a4eed9496f4343bd9a39f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aac4b65141a4eed9496f4343bd9a39f" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 42083 } }
I20260620 19:14:19.778555 31484 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4aac4b65141a4eed9496f4343bd9a39f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:14:19.778612 31484 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4aac4b65141a4eed9496f4343bd9a39f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aac4b65141a4eed9496f4343bd9a39f, State: Initialized, Role: FOLLOWER
I20260620 19:14:19.779129 31484 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4aac4b65141a4eed9496f4343bd9a39f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aac4b65141a4eed9496f4343bd9a39f" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 42083 } }
I20260620 19:14:19.779309 31484 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4aac4b65141a4eed9496f4343bd9a39f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:14:19.779381 31484 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4aac4b65141a4eed9496f4343bd9a39f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:14:19.779503 31484 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4aac4b65141a4eed9496f4343bd9a39f [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:14:19.780777 31484 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4aac4b65141a4eed9496f4343bd9a39f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aac4b65141a4eed9496f4343bd9a39f" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 42083 } }
I20260620 19:14:19.781174 31484 leader_election.cc:302] T 00000000000000000000000000000000 P 4aac4b65141a4eed9496f4343bd9a39f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4aac4b65141a4eed9496f4343bd9a39f; no voters:
I20260620 19:14:19.781549 31484 leader_election.cc:288] T 00000000000000000000000000000000 P 4aac4b65141a4eed9496f4343bd9a39f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:14:19.781675 31489 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4aac4b65141a4eed9496f4343bd9a39f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:14:19.781900 31489 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4aac4b65141a4eed9496f4343bd9a39f [term 1 LEADER]: Becoming Leader. State: Replica: 4aac4b65141a4eed9496f4343bd9a39f, State: Running, Role: LEADER
I20260620 19:14:19.782295 31489 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4aac4b65141a4eed9496f4343bd9a39f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aac4b65141a4eed9496f4343bd9a39f" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 42083 } }
I20260620 19:14:19.782985 31484 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4aac4b65141a4eed9496f4343bd9a39f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:14:19.784003 31491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4aac4b65141a4eed9496f4343bd9a39f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4aac4b65141a4eed9496f4343bd9a39f. Latest consensus state: current_term: 1 leader_uuid: "4aac4b65141a4eed9496f4343bd9a39f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aac4b65141a4eed9496f4343bd9a39f" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 42083 } } }
I20260620 19:14:19.784147 31491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4aac4b65141a4eed9496f4343bd9a39f [sys.catalog]: This master's current role is: LEADER
I20260620 19:14:19.784513 31490 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4aac4b65141a4eed9496f4343bd9a39f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4aac4b65141a4eed9496f4343bd9a39f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aac4b65141a4eed9496f4343bd9a39f" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 42083 } } }
I20260620 19:14:19.784574 31487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:19.769256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43869 (local address 127.9.155.126:42083)
0620 19:14:19.769730 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:19.769749 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:19.769793 (+ 44us) server_negotiation.cc:407] Connection header received
0620 19:14:19.770695 (+ 902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:19.770722 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:19.771171 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:19.771545 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:19.772441 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:19.773907 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:19.774793 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:19.775493 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:19.775666 (+ 173us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:19.778482 (+ 2816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:19.778489 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:19.778506 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:14:19.778544 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:19.780690 (+ 2146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:19.781288 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:19.781294 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:19.781302 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:19.781450 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:19.781902 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:19.781906 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:19.781907 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:19.782468 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:19.782712 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:19.783045 (+ 333us) server_negotiation.cc:299] Negotiation successful
0620 19:14:19.783486 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":127,"threads_started":1}
I20260620 19:14:19.784597 31490 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4aac4b65141a4eed9496f4343bd9a39f [sys.catalog]: This master's current role is: LEADER
I20260620 19:14:19.787988 9837 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260620 19:14:19.788354 31504 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4aac4b65141a4eed9496f4343bd9a39f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:14:19.788436 31504 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:14:19.788496 31498 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:14:19.789368 31498 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:14:19.796180 31498 catalog_manager.cc:1347] Generated new cluster ID: 75f401d6ebfd4a188cbe9e216a6d2376
I20260620 19:14:19.796278 31498 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:14:19.798305 31487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:19.790354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51479 (local address 127.9.155.126:42083)
0620 19:14:19.790462 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:19.790467 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:19.790654 (+ 187us) server_negotiation.cc:407] Connection header received
0620 19:14:19.790790 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:19.790793 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:19.790851 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:19.790931 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:19.791669 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:19.792200 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:19.793022 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:19.793295 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:19.793389 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:19.794911 (+ 1522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:19.794915 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:19.794917 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:14:19.794940 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:19.796386 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:19.796952 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:19.796956 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:19.796958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:19.797046 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:19.797384 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:19.797387 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:19.797388 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:19.797592 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:19.797696 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:19.798055 (+ 359us) server_negotiation.cc:299] Negotiation successful
0620 19:14:19.798154 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 19:14:19.804047 31498 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:14:19.805078 31498 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:14:19.815471 31498 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4aac4b65141a4eed9496f4343bd9a39f: Generated new TSK 0
I20260620 19:14:19.816267 31498 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:14:19.826145 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 31416
W20260620 19:14:19.830996 31401 connection.cc:535] client connection to 127.9.155.126:42083 recv error: Network error: failed to read from TLS socket (remote: 127.9.155.126:42083): Connection reset by peer (error 104)
2026-06-20T19:14:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1327 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982595231230-9837-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:14:19 dist-test-slave-n3fx krb5kdc[31515](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:14:19 dist-test-slave-n3fx krb5kdc[31515](info): set up 2 sockets
Jun 20 19:14:19 dist-test-slave-n3fx krb5kdc[31515](info): commencing operation
krb5kdc: starting...
W20260620 19:14:20.800140 9837 mini_kdc.cc:121] Time spent starting KDC: real 0.955s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982595231230-9837-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982595231230-9837-0/krb5kdc/test-user.keytab.
Jun 20 19:14:20 dist-test-slave-n3fx krb5kdc[31515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:14:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:14:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.155.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.155.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.155.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab.
Entry for principal kudu/127.9.155.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab.
I20260620 19:14:20.923054 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982595231230-9837-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982595231230-9837-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982595231230-9837-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:36495
--webserver_interface=127.9.155.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab
--principal=kudu/127.9.155.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:34317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.155.126:36495
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:14:21.032235 31528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:14:21.032516 31528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:14:21.032565 31528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:14:21.036587 31528 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:14:21.036655 31528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:14:21.036674 31528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:14:21.036688 31528 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:14:21.036701 31528 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:14:21.042186 31528 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:34317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982595231230-9837-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982595231230-9837-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.155.126:36495
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab
--principal=kudu/127.9.155.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:36495
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.155.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982595231230-9837-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:14:21.043277 31528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:14:21.044296 31528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:14:21.053356 31534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:14:21.053356 31536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:14:21.053403 31533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:14:21.053632 31528 server_base.cc:1034] running on GCE node
I20260620 19:14:21.054309 31528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:14:21.055480 31528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:14:21.056653 31528 hybrid_clock.cc:648] HybridClock initialized: now 1781982861056627 us; error 39 us; skew 500 ppm
Jun 20 19:14:21 dist-test-slave-n3fx krb5kdc[31515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982861, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:14:21.059593 31528 init.cc:377] Logged in from keytab as kudu/127.9.155.126@KRBTEST.COM (short username kudu)
I20260620 19:14:21.060693 31528 webserver.cc:460] Webserver started at http://127.9.155.126:36221/ using document root <none> and password file <none>
I20260620 19:14:21.061292 31528 fs_manager.cc:359] Metadata directory not provided
I20260620 19:14:21.061367 31528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:14:21.061581 31528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:14:21.063930 31528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982595231230-9837-0/minicluster-data/master-0/data/instance:
uuid: "1a09153f36a34a26bf67b039c278280e"
format_stamp: "Formatted at 2026-06-20 19:14:21 on dist-test-slave-n3fx"
server_key: "62b78e09884d34aa31e86e9b4abaec10"
server_key_iv: "45866f48316deaed32ae728ea6c05e69"
server_key_version: "encryptionkey@0"
I20260620 19:14:21.064580 31528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982595231230-9837-0/minicluster-data/master-0/wal/instance:
uuid: "1a09153f36a34a26bf67b039c278280e"
format_stamp: "Formatted at 2026-06-20 19:14:21 on dist-test-slave-n3fx"
server_key: "62b78e09884d34aa31e86e9b4abaec10"
server_key_iv: "45866f48316deaed32ae728ea6c05e69"
server_key_version: "encryptionkey@0"
I20260620 19:14:21.068514 31528 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 19:14:21.071280 31543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:21.072691 31528 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:14:21.072819 31528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982595231230-9837-0/minicluster-data/master-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982595231230-9837-0/minicluster-data/master-0/wal
uuid: "1a09153f36a34a26bf67b039c278280e"
format_stamp: "Formatted at 2026-06-20 19:14:21 on dist-test-slave-n3fx"
server_key: "62b78e09884d34aa31e86e9b4abaec10"
server_key_iv: "45866f48316deaed32ae728ea6c05e69"
server_key_version: "encryptionkey@0"
I20260620 19:14:21.072927 31528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982595231230-9837-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982595231230-9837-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982595231230-9837-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:14:21.084045 31528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:14:21.087178 31528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:14:21.087363 31528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:14:21.094861 31528 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.126:36495
I20260620 19:14:21.094905 31595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.126:36495 every 8 connection(s)
I20260620 19:14:21.095814 31528 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
I20260620 19:14:21.098562 31596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:14:21.099301 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 31528
I20260620 19:14:21.099443 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982595231230-9837-0/minicluster-data/master-0/wal/instance
I20260620 19:14:21.099740 9837 external_mini_cluster.cc:1428] Setting key 489da423a2671e801bc244b16090c63a
I20260620 19:14:21.105427 31596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a09153f36a34a26bf67b039c278280e: Bootstrap starting.
I20260620 19:14:21.108315 31596 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a09153f36a34a26bf67b039c278280e: Neither blocks nor log segments found. Creating new log.
I20260620 19:14:21.109210 31596 log.cc:824] T 00000000000000000000000000000000 P 1a09153f36a34a26bf67b039c278280e: Log is configured to *not* fsync() on all Append() calls
I20260620 19:14:21.109851 31599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:21.101145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52465 (local address 127.9.155.126:36495)
0620 19:14:21.101583 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:21.101603 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:21.101647 (+ 44us) server_negotiation.cc:407] Connection header received
0620 19:14:21.102457 (+ 810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:21.102482 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:21.102911 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:21.103284 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:21.103683 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:21.105119 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:21.106082 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:21.106791 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:21.106978 (+ 187us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:21.107823 (+ 845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:21.107830 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:21.107847 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:14:21.108390 (+ 543us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:21.108461 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:14:21.108493 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:21.108634 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:21.108817 (+ 183us) server_negotiation.cc:299] Negotiation successful
0620 19:14:21.108950 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":117,"threads_started":1}
I20260620 19:14:21.111276 31596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a09153f36a34a26bf67b039c278280e: No bootstrap required, opened a new log
I20260620 19:14:21.112758 9837 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:14:21.115748 31596 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a09153f36a34a26bf67b039c278280e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a09153f36a34a26bf67b039c278280e" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 36495 } }
I20260620 19:14:21.116057 31596 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a09153f36a34a26bf67b039c278280e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:14:21.116099 31596 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a09153f36a34a26bf67b039c278280e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a09153f36a34a26bf67b039c278280e, State: Initialized, Role: FOLLOWER
I20260620 19:14:21.116568 31596 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a09153f36a34a26bf67b039c278280e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a09153f36a34a26bf67b039c278280e" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 36495 } }
I20260620 19:14:21.116722 31596 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a09153f36a34a26bf67b039c278280e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:14:21.116796 31596 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a09153f36a34a26bf67b039c278280e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:14:21.116900 31596 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a09153f36a34a26bf67b039c278280e [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:14:21.118072 31596 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a09153f36a34a26bf67b039c278280e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a09153f36a34a26bf67b039c278280e" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 36495 } }
I20260620 19:14:21.118474 31596 leader_election.cc:302] T 00000000000000000000000000000000 P 1a09153f36a34a26bf67b039c278280e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a09153f36a34a26bf67b039c278280e; no voters:
I20260620 19:14:21.118806 31596 leader_election.cc:288] T 00000000000000000000000000000000 P 1a09153f36a34a26bf67b039c278280e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:14:21.118970 31606 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a09153f36a34a26bf67b039c278280e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:14:21.119156 31606 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a09153f36a34a26bf67b039c278280e [term 1 LEADER]: Becoming Leader. State: Replica: 1a09153f36a34a26bf67b039c278280e, State: Running, Role: LEADER
I20260620 19:14:21.119488 31606 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a09153f36a34a26bf67b039c278280e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a09153f36a34a26bf67b039c278280e" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 36495 } }
I20260620 19:14:21.119769 31599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:21.115100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53067 (local address 127.9.155.126:36495)
0620 19:14:21.115228 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:21.115232 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:21.115382 (+ 150us) server_negotiation.cc:407] Connection header received
0620 19:14:21.115468 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:21.115470 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:21.115512 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:21.115581 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:21.115913 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:21.116578 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:21.117747 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:21.118053 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:21.118134 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:21.119124 (+ 990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:21.119128 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:21.119131 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:14:21.119145 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:21.119174 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:14:21.119188 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:21.119310 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:21.119597 (+ 287us) server_negotiation.cc:299] Negotiation successful
0620 19:14:21.119640 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 19:14:21.120312 31596 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a09153f36a34a26bf67b039c278280e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:14:21.121141 31607 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a09153f36a34a26bf67b039c278280e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a09153f36a34a26bf67b039c278280e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a09153f36a34a26bf67b039c278280e" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 36495 } } }
I20260620 19:14:21.121271 31607 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a09153f36a34a26bf67b039c278280e [sys.catalog]: This master's current role is: LEADER
I20260620 19:14:21.121650 31615 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:14:21.121626 31608 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a09153f36a34a26bf67b039c278280e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a09153f36a34a26bf67b039c278280e. Latest consensus state: current_term: 1 leader_uuid: "1a09153f36a34a26bf67b039c278280e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a09153f36a34a26bf67b039c278280e" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 36495 } } }
I20260620 19:14:21.121716 31608 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a09153f36a34a26bf67b039c278280e [sys.catalog]: This master's current role is: LEADER
I20260620 19:14:21.124495 31615 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:14:21.130005 31615 catalog_manager.cc:1347] Generated new cluster ID: be5c9011dd42416baf915a0430bedba1
I20260620 19:14:21.130085 31615 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:14:21.141464 31615 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:14:21.142340 31615 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:14:21.151439 31615 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a09153f36a34a26bf67b039c278280e: Generated new TSK 0
I20260620 19:14:21.152243 31615 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:14:21.159420 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 31528
W20260620 19:14:21.163993 31513 connection.cc:535] client connection to 127.9.155.126:36495 recv error: Network error: failed to read from TLS socket (remote: 127.9.155.126:36495): Connection reset by peer (error 104)
2026-06-20T19:14:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1333 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982595231230-9837-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:14:21 dist-test-slave-n3fx krb5kdc[31627](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:14:21 dist-test-slave-n3fx krb5kdc[31627](info): set up 2 sockets
Jun 20 19:14:21 dist-test-slave-n3fx krb5kdc[31627](info): commencing operation
krb5kdc: starting...
W20260620 19:14:22.108273 9837 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982595231230-9837-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982595231230-9837-0/krb5kdc/test-user.keytab.
Jun 20 19:14:22 dist-test-slave-n3fx krb5kdc[31627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982862, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:14:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:14:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.155.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.155.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.155.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab.
Entry for principal kudu/127.9.155.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab.
I20260620 19:14:22.230947 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982595231230-9837-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982595231230-9837-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982595231230-9837-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:40875
--webserver_interface=127.9.155.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab
--principal=kudu/127.9.155.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:46593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.155.126:40875
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:14:22.345239 31640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:14:22.345541 31640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:14:22.345616 31640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:14:22.349833 31640 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:14:22.349915 31640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:14:22.349942 31640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:14:22.349964 31640 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:14:22.349985 31640 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:14:22.355648 31640 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:46593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982595231230-9837-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982595231230-9837-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.155.126:40875
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab
--principal=kudu/127.9.155.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:40875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.155.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982595231230-9837-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:14:22.356768 31640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:14:22.357760 31640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:14:22.366290 31648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:14:22.366290 31645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:14:22.366360 31640 server_base.cc:1034] running on GCE node
W20260620 19:14:22.366290 31646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:14:22.366984 31640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:14:22.368081 31640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:14:22.369242 31640 hybrid_clock.cc:648] HybridClock initialized: now 1781982862369224 us; error 34 us; skew 500 ppm
Jun 20 19:14:22 dist-test-slave-n3fx krb5kdc[31627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982862, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:14:22.371847 31640 init.cc:377] Logged in from keytab as kudu/127.9.155.126@KRBTEST.COM (short username kudu)
I20260620 19:14:22.372750 31640 webserver.cc:460] Webserver started at http://127.9.155.126:44513/ using document root <none> and password file <none>
I20260620 19:14:22.373279 31640 fs_manager.cc:359] Metadata directory not provided
I20260620 19:14:22.373343 31640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:14:22.373521 31640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:14:22.375509 31640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982595231230-9837-0/minicluster-data/master-0/data/instance:
uuid: "ca7757ae51534ed697af19bd2a1411b5"
format_stamp: "Formatted at 2026-06-20 19:14:22 on dist-test-slave-n3fx"
server_key: "c08f918e83dccf611393869e18674b02"
server_key_iv: "17ddb22be8effde00787051f96a5b5a4"
server_key_version: "encryptionkey@0"
I20260620 19:14:22.376034 31640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982595231230-9837-0/minicluster-data/master-0/wal/instance:
uuid: "ca7757ae51534ed697af19bd2a1411b5"
format_stamp: "Formatted at 2026-06-20 19:14:22 on dist-test-slave-n3fx"
server_key: "c08f918e83dccf611393869e18674b02"
server_key_iv: "17ddb22be8effde00787051f96a5b5a4"
server_key_version: "encryptionkey@0"
I20260620 19:14:22.379683 31640 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 19:14:22.382098 31655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:22.383383 31640 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 19:14:22.383512 31640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982595231230-9837-0/minicluster-data/master-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982595231230-9837-0/minicluster-data/master-0/wal
uuid: "ca7757ae51534ed697af19bd2a1411b5"
format_stamp: "Formatted at 2026-06-20 19:14:22 on dist-test-slave-n3fx"
server_key: "c08f918e83dccf611393869e18674b02"
server_key_iv: "17ddb22be8effde00787051f96a5b5a4"
server_key_version: "encryptionkey@0"
I20260620 19:14:22.383618 31640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982595231230-9837-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982595231230-9837-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982595231230-9837-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:14:22.388862 31640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:14:22.391955 31640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:14:22.392143 31640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:14:22.398888 31640 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.126:40875
I20260620 19:14:22.398916 31707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.126:40875 every 8 connection(s)
I20260620 19:14:22.399875 31640 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
I20260620 19:14:22.402670 31708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:14:22.407248 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 31640
I20260620 19:14:22.407374 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982595231230-9837-0/minicluster-data/master-0/wal/instance
I20260620 19:14:22.407712 9837 external_mini_cluster.cc:1428] Setting key eaa5bba4a9f6e54b39b9acb4324d6128
I20260620 19:14:22.408615 31708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca7757ae51534ed697af19bd2a1411b5: Bootstrap starting.
I20260620 19:14:22.411041 31708 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca7757ae51534ed697af19bd2a1411b5: Neither blocks nor log segments found. Creating new log.
I20260620 19:14:22.411870 31708 log.cc:824] T 00000000000000000000000000000000 P ca7757ae51534ed697af19bd2a1411b5: Log is configured to *not* fsync() on all Append() calls
I20260620 19:14:22.413071 31711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:22.409116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33055 (local address 127.9.155.126:40875)
0620 19:14:22.409670 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:22.409686 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:22.409726 (+ 40us) server_negotiation.cc:407] Connection header received
0620 19:14:22.410399 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:22.410416 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:22.410709 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:22.411026 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:22.411784 (+ 758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:22.411789 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:22.411800 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:14:22.411810 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:22.411866 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:14:22.411886 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:22.411987 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:22.412094 (+ 107us) server_negotiation.cc:299] Negotiation successful
0620 19:14:22.412285 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":145,"threads_started":1}
I20260620 19:14:22.413849 31708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca7757ae51534ed697af19bd2a1411b5: No bootstrap required, opened a new log
I20260620 19:14:22.415628 9837 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:14:22.417656 31708 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca7757ae51534ed697af19bd2a1411b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7757ae51534ed697af19bd2a1411b5" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 40875 } }
I20260620 19:14:22.417917 31708 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca7757ae51534ed697af19bd2a1411b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:14:22.417974 31708 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca7757ae51534ed697af19bd2a1411b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca7757ae51534ed697af19bd2a1411b5, State: Initialized, Role: FOLLOWER
I20260620 19:14:22.418454 31708 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca7757ae51534ed697af19bd2a1411b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7757ae51534ed697af19bd2a1411b5" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 40875 } }
I20260620 19:14:22.418632 31708 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca7757ae51534ed697af19bd2a1411b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:14:22.418716 31708 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca7757ae51534ed697af19bd2a1411b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:14:22.418885 31708 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca7757ae51534ed697af19bd2a1411b5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:14:22.419198 31711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:22.417492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40331 (local address 127.9.155.126:40875)
0620 19:14:22.417593 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:22.417604 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:22.417777 (+ 173us) server_negotiation.cc:407] Connection header received
0620 19:14:22.417892 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:22.417895 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:22.417927 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:22.417990 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:22.418799 (+ 809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:22.418802 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:22.418805 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:14:22.418818 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:22.418849 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:14:22.418862 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:22.418934 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:22.419026 (+ 92us) server_negotiation.cc:299] Negotiation successful
0620 19:14:22.419066 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 19:14:22.420145 31708 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca7757ae51534ed697af19bd2a1411b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7757ae51534ed697af19bd2a1411b5" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 40875 } }
I20260620 19:14:22.420518 31708 leader_election.cc:302] T 00000000000000000000000000000000 P ca7757ae51534ed697af19bd2a1411b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca7757ae51534ed697af19bd2a1411b5; no voters:
I20260620 19:14:22.420809 31708 leader_election.cc:288] T 00000000000000000000000000000000 P ca7757ae51534ed697af19bd2a1411b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:14:22.420909 31718 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca7757ae51534ed697af19bd2a1411b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:14:22.421108 31718 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca7757ae51534ed697af19bd2a1411b5 [term 1 LEADER]: Becoming Leader. State: Replica: ca7757ae51534ed697af19bd2a1411b5, State: Running, Role: LEADER
I20260620 19:14:22.421459 31718 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca7757ae51534ed697af19bd2a1411b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7757ae51534ed697af19bd2a1411b5" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 40875 } }
I20260620 19:14:22.422004 31708 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca7757ae51534ed697af19bd2a1411b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:14:22.423044 31719 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca7757ae51534ed697af19bd2a1411b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca7757ae51534ed697af19bd2a1411b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7757ae51534ed697af19bd2a1411b5" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 40875 } } }
I20260620 19:14:22.423213 31719 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca7757ae51534ed697af19bd2a1411b5 [sys.catalog]: This master's current role is: LEADER
I20260620 19:14:22.423038 31720 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca7757ae51534ed697af19bd2a1411b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca7757ae51534ed697af19bd2a1411b5. Latest consensus state: current_term: 1 leader_uuid: "ca7757ae51534ed697af19bd2a1411b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7757ae51534ed697af19bd2a1411b5" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 40875 } } }
I20260620 19:14:22.423451 31720 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca7757ae51534ed697af19bd2a1411b5 [sys.catalog]: This master's current role is: LEADER
I20260620 19:14:22.423601 31727 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:14:22.426721 31727 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:14:22.432476 31727 catalog_manager.cc:1347] Generated new cluster ID: 05268becb59341fd9db3a553e869e102
I20260620 19:14:22.432548 31727 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:14:22.451483 31727 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:14:22.451871 31727 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:14:22.459190 31727 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca7757ae51534ed697af19bd2a1411b5: Generated new TSK 0
I20260620 19:14:22.460016 31727 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:14:22.465890 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 31640
2026-06-20T19:14:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1306 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982595231230-9837-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:14:22 dist-test-slave-n3fx krb5kdc[31739](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:14:22 dist-test-slave-n3fx krb5kdc[31739](info): set up 2 sockets
Jun 20 19:14:22 dist-test-slave-n3fx krb5kdc[31739](info): commencing operation
krb5kdc: starting...
W20260620 19:14:23.428337 9837 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982595231230-9837-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982595231230-9837-0/krb5kdc/test-user.keytab.
Jun 20 19:14:23 dist-test-slave-n3fx krb5kdc[31739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:14:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:14:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.155.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.155.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.155.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab.
Entry for principal kudu/127.9.155.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab.
I20260620 19:14:23.548698 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982595231230-9837-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982595231230-9837-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982595231230-9837-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:33541
--webserver_interface=127.9.155.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab
--principal=kudu/127.9.155.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:46181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.155.126:33541
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:14:23.657030 31752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:14:23.657332 31752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:14:23.657392 31752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:14:23.661470 31752 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:14:23.661542 31752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:14:23.661569 31752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:14:23.661590 31752 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:14:23.661612 31752 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:14:23.666968 31752 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:46181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982595231230-9837-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982595231230-9837-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.155.126:33541
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab
--principal=kudu/127.9.155.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:33541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.155.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31752
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982595231230-9837-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:14:23.668090 31752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:14:23.669166 31752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:14:23.677596 31758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:14:23.677698 31752 server_base.cc:1034] running on GCE node
W20260620 19:14:23.677604 31760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:14:23.677610 31757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:14:23.678408 31752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:14:23.679538 31752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:14:23.680771 31752 hybrid_clock.cc:648] HybridClock initialized: now 1781982863680754 us; error 30 us; skew 500 ppm
Jun 20 19:14:23 dist-test-slave-n3fx krb5kdc[31739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982863, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:14:23.683435 31752 init.cc:377] Logged in from keytab as kudu/127.9.155.126@KRBTEST.COM (short username kudu)
I20260620 19:14:23.684417 31752 webserver.cc:460] Webserver started at http://127.9.155.126:42641/ using document root <none> and password file <none>
I20260620 19:14:23.684954 31752 fs_manager.cc:359] Metadata directory not provided
I20260620 19:14:23.685014 31752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:14:23.685215 31752 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:14:23.687259 31752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982595231230-9837-0/minicluster-data/master-0/data/instance:
uuid: "4beba387772144f3b4a1836a1a73a726"
format_stamp: "Formatted at 2026-06-20 19:14:23 on dist-test-slave-n3fx"
server_key: "7ac3ea5ce082b1397af58b36a1fe2d51"
server_key_iv: "2cc3a376bf8da8bda26224f118d56bda"
server_key_version: "encryptionkey@0"
I20260620 19:14:23.687809 31752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982595231230-9837-0/minicluster-data/master-0/wal/instance:
uuid: "4beba387772144f3b4a1836a1a73a726"
format_stamp: "Formatted at 2026-06-20 19:14:23 on dist-test-slave-n3fx"
server_key: "7ac3ea5ce082b1397af58b36a1fe2d51"
server_key_iv: "2cc3a376bf8da8bda26224f118d56bda"
server_key_version: "encryptionkey@0"
I20260620 19:14:23.691502 31752 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 19:14:23.694036 31767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:23.695355 31752 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 19:14:23.695474 31752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982595231230-9837-0/minicluster-data/master-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982595231230-9837-0/minicluster-data/master-0/wal
uuid: "4beba387772144f3b4a1836a1a73a726"
format_stamp: "Formatted at 2026-06-20 19:14:23 on dist-test-slave-n3fx"
server_key: "7ac3ea5ce082b1397af58b36a1fe2d51"
server_key_iv: "2cc3a376bf8da8bda26224f118d56bda"
server_key_version: "encryptionkey@0"
I20260620 19:14:23.695588 31752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982595231230-9837-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982595231230-9837-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982595231230-9837-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:14:23.709122 31752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:14:23.712447 31752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:14:23.712623 31752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:14:23.719774 31752 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.126:33541
I20260620 19:14:23.719834 31819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.126:33541 every 8 connection(s)
I20260620 19:14:23.720944 31752 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
I20260620 19:14:23.724027 31820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:14:23.724858 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 31752
I20260620 19:14:23.725020 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982595231230-9837-0/minicluster-data/master-0/wal/instance
I20260620 19:14:23.725358 9837 external_mini_cluster.cc:1428] Setting key 50e9c076caa89b1350dfa11c8bd4077b
I20260620 19:14:23.730441 31820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4beba387772144f3b4a1836a1a73a726: Bootstrap starting.
I20260620 19:14:23.733093 31820 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4beba387772144f3b4a1836a1a73a726: Neither blocks nor log segments found. Creating new log.
I20260620 19:14:23.733897 31820 log.cc:824] T 00000000000000000000000000000000 P 4beba387772144f3b4a1836a1a73a726: Log is configured to *not* fsync() on all Append() calls
Jun 20 19:14:23 dist-test-slave-n3fx krb5kdc[31739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:14:23.736081 31820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4beba387772144f3b4a1836a1a73a726: No bootstrap required, opened a new log
I20260620 19:14:23.740182 31823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:23.726835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36099 (local address 127.9.155.126:33541)
0620 19:14:23.727312 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:23.727331 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:23.727516 (+ 185us) server_negotiation.cc:407] Connection header received
0620 19:14:23.728382 (+ 866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:23.728402 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:23.728788 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:23.729135 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:23.729961 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:23.731058 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:23.732167 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:23.732726 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:23.732875 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:23.735703 (+ 2828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:23.735709 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:23.735723 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:14:23.735752 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:23.737455 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:23.737941 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:23.737946 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:23.737950 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:23.738058 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:23.738327 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:23.738329 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:23.738330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:23.738699 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:23.738842 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:23.739124 (+ 282us) server_negotiation.cc:299] Negotiation successful
0620 19:14:23.739384 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":126,"threads_started":1}
I20260620 19:14:23.740080 31820 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4beba387772144f3b4a1836a1a73a726 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4beba387772144f3b4a1836a1a73a726" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 33541 } }
I20260620 19:14:23.740350 31820 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4beba387772144f3b4a1836a1a73a726 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:14:23.740406 31820 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4beba387772144f3b4a1836a1a73a726 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4beba387772144f3b4a1836a1a73a726, State: Initialized, Role: FOLLOWER
I20260620 19:14:23.740846 31820 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4beba387772144f3b4a1836a1a73a726 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4beba387772144f3b4a1836a1a73a726" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 33541 } }
I20260620 19:14:23.740970 31820 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4beba387772144f3b4a1836a1a73a726 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:14:23.741029 31820 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4beba387772144f3b4a1836a1a73a726 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:14:23.741132 31820 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4beba387772144f3b4a1836a1a73a726 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:14:23.742267 31820 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4beba387772144f3b4a1836a1a73a726 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4beba387772144f3b4a1836a1a73a726" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 33541 } }
I20260620 19:14:23.742611 31820 leader_election.cc:302] T 00000000000000000000000000000000 P 4beba387772144f3b4a1836a1a73a726 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4beba387772144f3b4a1836a1a73a726; no voters:
I20260620 19:14:23.742937 31820 leader_election.cc:288] T 00000000000000000000000000000000 P 4beba387772144f3b4a1836a1a73a726 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:14:23.743045 9837 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:14:23.743043 31825 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4beba387772144f3b4a1836a1a73a726 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:14:23.743251 31825 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4beba387772144f3b4a1836a1a73a726 [term 1 LEADER]: Becoming Leader. State: Replica: 4beba387772144f3b4a1836a1a73a726, State: Running, Role: LEADER
I20260620 19:14:23.743628 31825 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4beba387772144f3b4a1836a1a73a726 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4beba387772144f3b4a1836a1a73a726" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 33541 } }
I20260620 19:14:23.744067 31820 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4beba387772144f3b4a1836a1a73a726 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:14:23.745267 31826 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4beba387772144f3b4a1836a1a73a726 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4beba387772144f3b4a1836a1a73a726" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4beba387772144f3b4a1836a1a73a726" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 33541 } } }
I20260620 19:14:23.745397 31826 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4beba387772144f3b4a1836a1a73a726 [sys.catalog]: This master's current role is: LEADER
I20260620 19:14:23.745710 31827 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4beba387772144f3b4a1836a1a73a726 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4beba387772144f3b4a1836a1a73a726. Latest consensus state: current_term: 1 leader_uuid: "4beba387772144f3b4a1836a1a73a726" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4beba387772144f3b4a1836a1a73a726" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 33541 } } }
I20260620 19:14:23.745815 31827 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4beba387772144f3b4a1836a1a73a726 [sys.catalog]: This master's current role is: LEADER
I20260620 19:14:23.745769 31839 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:14:23.749161 31839 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:14:23.754504 31823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:23.746841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49765 (local address 127.9.155.126:33541)
0620 19:14:23.746945 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:23.746949 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:23.747280 (+ 331us) server_negotiation.cc:407] Connection header received
0620 19:14:23.747392 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:23.747395 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:23.747439 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:23.747502 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:23.748148 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:23.748708 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:23.749518 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:23.749860 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:23.749950 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:23.751268 (+ 1318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:23.751272 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:23.751274 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:14:23.751299 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:23.752967 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:23.753401 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:23.753404 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:23.753406 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:23.753484 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:23.753722 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:23.753725 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:23.753727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:23.753900 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:23.754026 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:23.754237 (+ 211us) server_negotiation.cc:299] Negotiation successful
0620 19:14:23.754349 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 19:14:23.755579 31839 catalog_manager.cc:1347] Generated new cluster ID: 9c94850d19834ff0b7fe1d38a30c0633
I20260620 19:14:23.755653 31839 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:14:23.771001 31839 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:14:23.772382 31839 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:14:23.781454 31839 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4beba387772144f3b4a1836a1a73a726: Generated new TSK 0
I20260620 19:14:23.782364 31839 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:14:23.791018 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 31752
2026-06-20T19:14:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1324 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982595231230-9837-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:14:23 dist-test-slave-n3fx krb5kdc[31851](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:14:23 dist-test-slave-n3fx krb5kdc[31851](info): set up 2 sockets
Jun 20 19:14:23 dist-test-slave-n3fx krb5kdc[31851](info): commencing operation
krb5kdc: starting...
W20260620 19:14:24.768357 9837 mini_kdc.cc:121] Time spent starting KDC: real 0.959s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982595231230-9837-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982595231230-9837-0/krb5kdc/test-user.keytab.
Jun 20 19:14:24 dist-test-slave-n3fx krb5kdc[31851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:14:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:14:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.155.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.155.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.155.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab.
Entry for principal kudu/127.9.155.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab.
I20260620 19:14:24.892148 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982595231230-9837-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982595231230-9837-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982595231230-9837-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:46223
--webserver_interface=127.9.155.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab
--principal=kudu/127.9.155.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:40019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.155.126:46223
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:14:25.001056 31864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:14:25.001353 31864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:14:25.001415 31864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:14:25.005550 31864 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:14:25.005641 31864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:14:25.005669 31864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:14:25.005692 31864 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:14:25.005829 31864 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:14:25.012418 31864 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:40019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982595231230-9837-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982595231230-9837-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.155.126:46223
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab
--principal=kudu/127.9.155.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:46223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.155.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31864
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982595231230-9837-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:14:25.013639 31864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:14:25.014748 31864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:14:25.023938 31869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:14:25.023914 31870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:14:25.024294 31864 server_base.cc:1034] running on GCE node
W20260620 19:14:25.023914 31872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:14:25.024947 31864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:14:25.026144 31864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:14:25.027319 31864 hybrid_clock.cc:648] HybridClock initialized: now 1781982865027292 us; error 43 us; skew 500 ppm
Jun 20 19:14:25 dist-test-slave-n3fx krb5kdc[31851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982865, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:14:25.033561 31864 init.cc:377] Logged in from keytab as kudu/127.9.155.126@KRBTEST.COM (short username kudu)
I20260620 19:14:25.034605 31864 webserver.cc:460] Webserver started at http://127.9.155.126:39093/ using document root <none> and password file <none>
I20260620 19:14:25.035182 31864 fs_manager.cc:359] Metadata directory not provided
I20260620 19:14:25.035254 31864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:14:25.035459 31864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:14:25.037573 31864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982595231230-9837-0/minicluster-data/master-0/data/instance:
uuid: "e01c099e9fad4362b705dba223bd1ea5"
format_stamp: "Formatted at 2026-06-20 19:14:25 on dist-test-slave-n3fx"
server_key: "fa9b300c026437afe0231e73aba0a97e"
server_key_iv: "7828fcaf9bb65a8a0a9ccdb07273f126"
server_key_version: "encryptionkey@0"
I20260620 19:14:25.038102 31864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982595231230-9837-0/minicluster-data/master-0/wal/instance:
uuid: "e01c099e9fad4362b705dba223bd1ea5"
format_stamp: "Formatted at 2026-06-20 19:14:25 on dist-test-slave-n3fx"
server_key: "fa9b300c026437afe0231e73aba0a97e"
server_key_iv: "7828fcaf9bb65a8a0a9ccdb07273f126"
server_key_version: "encryptionkey@0"
I20260620 19:14:25.041877 31864 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 19:14:25.044667 31879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:25.046185 31864 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:14:25.046330 31864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982595231230-9837-0/minicluster-data/master-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982595231230-9837-0/minicluster-data/master-0/wal
uuid: "e01c099e9fad4362b705dba223bd1ea5"
format_stamp: "Formatted at 2026-06-20 19:14:25 on dist-test-slave-n3fx"
server_key: "fa9b300c026437afe0231e73aba0a97e"
server_key_iv: "7828fcaf9bb65a8a0a9ccdb07273f126"
server_key_version: "encryptionkey@0"
I20260620 19:14:25.046455 31864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982595231230-9837-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982595231230-9837-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982595231230-9837-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:14:25.063771 31864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:14:25.067284 31864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:14:25.067492 31864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:14:25.074815 31864 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.126:46223
I20260620 19:14:25.074875 31931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.126:46223 every 8 connection(s)
I20260620 19:14:25.075804 31864 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
I20260620 19:14:25.078558 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 31864
I20260620 19:14:25.078684 31932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:14:25.078742 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982595231230-9837-0/minicluster-data/master-0/wal/instance
I20260620 19:14:25.079038 9837 external_mini_cluster.cc:1428] Setting key d0b11a26284e1d85ca093459818a8354
I20260620 19:14:25.085561 31932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e01c099e9fad4362b705dba223bd1ea5: Bootstrap starting.
I20260620 19:14:25.088337 31932 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e01c099e9fad4362b705dba223bd1ea5: Neither blocks nor log segments found. Creating new log.
I20260620 19:14:25.088896 31935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:25.080454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43539 (local address 127.9.155.126:46223)
0620 19:14:25.080924 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:25.080939 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:25.080975 (+ 36us) server_negotiation.cc:407] Connection header received
0620 19:14:25.081895 (+ 920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:25.081917 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:25.082307 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:25.082647 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:25.083058 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:25.084387 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:25.085327 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:25.085922 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:25.086100 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:25.087030 (+ 930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:25.087036 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:25.087049 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:14:25.087474 (+ 425us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:25.087535 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:14:25.087559 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:25.087708 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:25.087836 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 19:14:25.088040 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":164,"threads_started":1}
I20260620 19:14:25.089094 31932 log.cc:824] T 00000000000000000000000000000000 P e01c099e9fad4362b705dba223bd1ea5: Log is configured to *not* fsync() on all Append() calls
I20260620 19:14:25.091123 31932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e01c099e9fad4362b705dba223bd1ea5: No bootstrap required, opened a new log
I20260620 19:14:25.091650 9837 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:14:25.095480 31932 raft_consensus.cc:348] T 00000000000000000000000000000000 P e01c099e9fad4362b705dba223bd1ea5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e01c099e9fad4362b705dba223bd1ea5" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 46223 } }
I20260620 19:14:25.095788 31932 raft_consensus.cc:374] T 00000000000000000000000000000000 P e01c099e9fad4362b705dba223bd1ea5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:14:25.095846 31932 raft_consensus.cc:729] T 00000000000000000000000000000000 P e01c099e9fad4362b705dba223bd1ea5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e01c099e9fad4362b705dba223bd1ea5, State: Initialized, Role: FOLLOWER
I20260620 19:14:25.096360 31932 consensus_queue.cc:260] T 00000000000000000000000000000000 P e01c099e9fad4362b705dba223bd1ea5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e01c099e9fad4362b705dba223bd1ea5" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 46223 } }
I20260620 19:14:25.096499 31932 raft_consensus.cc:388] T 00000000000000000000000000000000 P e01c099e9fad4362b705dba223bd1ea5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:14:25.096560 31932 raft_consensus.cc:482] T 00000000000000000000000000000000 P e01c099e9fad4362b705dba223bd1ea5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:14:25.096657 31932 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e01c099e9fad4362b705dba223bd1ea5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:14:25.097587 31932 raft_consensus.cc:504] T 00000000000000000000000000000000 P e01c099e9fad4362b705dba223bd1ea5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e01c099e9fad4362b705dba223bd1ea5" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 46223 } }
I20260620 19:14:25.097862 31935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:25.093888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43725 (local address 127.9.155.126:46223)
0620 19:14:25.094005 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:25.094010 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:25.094189 (+ 179us) server_negotiation.cc:407] Connection header received
0620 19:14:25.094304 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:25.094306 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:25.094349 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:25.094418 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:25.094742 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:25.095251 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:25.096319 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:25.096607 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:25.096709 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:25.097450 (+ 741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:25.097453 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:25.097456 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:14:25.097469 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:25.097496 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:14:25.097506 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:25.097564 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:25.097677 (+ 113us) server_negotiation.cc:299] Negotiation successful
0620 19:14:25.097730 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 19:14:25.097980 31932 leader_election.cc:302] T 00000000000000000000000000000000 P e01c099e9fad4362b705dba223bd1ea5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e01c099e9fad4362b705dba223bd1ea5; no voters:
I20260620 19:14:25.098366 31932 leader_election.cc:288] T 00000000000000000000000000000000 P e01c099e9fad4362b705dba223bd1ea5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:14:25.098454 31942 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e01c099e9fad4362b705dba223bd1ea5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:14:25.098675 31942 raft_consensus.cc:686] T 00000000000000000000000000000000 P e01c099e9fad4362b705dba223bd1ea5 [term 1 LEADER]: Becoming Leader. State: Replica: e01c099e9fad4362b705dba223bd1ea5, State: Running, Role: LEADER
I20260620 19:14:25.099025 31942 consensus_queue.cc:237] T 00000000000000000000000000000000 P e01c099e9fad4362b705dba223bd1ea5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e01c099e9fad4362b705dba223bd1ea5" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 46223 } }
I20260620 19:14:25.099618 31932 sys_catalog.cc:564] T 00000000000000000000000000000000 P e01c099e9fad4362b705dba223bd1ea5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:14:25.100126 31942 sys_catalog.cc:455] T 00000000000000000000000000000000 P e01c099e9fad4362b705dba223bd1ea5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e01c099e9fad4362b705dba223bd1ea5. Latest consensus state: current_term: 1 leader_uuid: "e01c099e9fad4362b705dba223bd1ea5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e01c099e9fad4362b705dba223bd1ea5" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 46223 } } }
I20260620 19:14:25.100329 31942 sys_catalog.cc:458] T 00000000000000000000000000000000 P e01c099e9fad4362b705dba223bd1ea5 [sys.catalog]: This master's current role is: LEADER
I20260620 19:14:25.100636 31943 sys_catalog.cc:455] T 00000000000000000000000000000000 P e01c099e9fad4362b705dba223bd1ea5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e01c099e9fad4362b705dba223bd1ea5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e01c099e9fad4362b705dba223bd1ea5" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 46223 } } }
I20260620 19:14:25.100773 31943 sys_catalog.cc:458] T 00000000000000000000000000000000 P e01c099e9fad4362b705dba223bd1ea5 [sys.catalog]: This master's current role is: LEADER
I20260620 19:14:25.100878 31949 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:14:25.103853 31949 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:14:25.109279 31949 catalog_manager.cc:1347] Generated new cluster ID: a1b2cdb88f3647e4ba1b62479838119d
I20260620 19:14:25.109368 31949 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:14:25.148898 31949 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:14:25.150285 31949 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:14:25.156245 31949 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e01c099e9fad4362b705dba223bd1ea5: Generated new TSK 0
I20260620 19:14:25.157059 31949 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:14:25.166563 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 31864
2026-06-20T19:14:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1375 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982595231230-9837-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:14:25 dist-test-slave-n3fx krb5kdc[31963](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:14:25 dist-test-slave-n3fx krb5kdc[31963](info): set up 2 sockets
Jun 20 19:14:25 dist-test-slave-n3fx krb5kdc[31963](info): commencing operation
krb5kdc: starting...
W20260620 19:14:26.128425 9837 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982595231230-9837-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982595231230-9837-0/krb5kdc/test-user.keytab.
Jun 20 19:14:26 dist-test-slave-n3fx krb5kdc[31963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982866, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:14:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:14:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.155.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.155.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.155.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab.
Entry for principal kudu/127.9.155.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab.
I20260620 19:14:26.246268 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982595231230-9837-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982595231230-9837-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982595231230-9837-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:42745
--webserver_interface=127.9.155.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab
--principal=kudu/127.9.155.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:40303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.155.126:42745
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:14:26.357352 31976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:14:26.357617 31976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:14:26.357659 31976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:14:26.361765 31976 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:14:26.361831 31976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:14:26.361850 31976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:14:26.361865 31976 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:14:26.361878 31976 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:14:26.367230 31976 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:40303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982595231230-9837-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982595231230-9837-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.155.126:42745
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab
--principal=kudu/127.9.155.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:42745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.155.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31976
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982595231230-9837-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:14:26.368265 31976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:14:26.369211 31976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:14:26.377287 31982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:14:26.377331 31984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:14:26.377357 31981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:14:26.377602 31976 server_base.cc:1034] running on GCE node
I20260620 19:14:26.378226 31976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:14:26.379302 31976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:14:26.380481 31976 hybrid_clock.cc:648] HybridClock initialized: now 1781982866380465 us; error 32 us; skew 500 ppm
Jun 20 19:14:26 dist-test-slave-n3fx krb5kdc[31963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982866, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:14:26.383203 31976 init.cc:377] Logged in from keytab as kudu/127.9.155.126@KRBTEST.COM (short username kudu)
I20260620 19:14:26.384099 31976 webserver.cc:460] Webserver started at http://127.9.155.126:34511/ using document root <none> and password file <none>
I20260620 19:14:26.384684 31976 fs_manager.cc:359] Metadata directory not provided
I20260620 19:14:26.384752 31976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:14:26.384995 31976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:14:26.387087 31976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982595231230-9837-0/minicluster-data/master-0/data/instance:
uuid: "382f3ca60c98404686ba31891d46e66c"
format_stamp: "Formatted at 2026-06-20 19:14:26 on dist-test-slave-n3fx"
server_key: "2a57e115bb980d86ef924a43e1668959"
server_key_iv: "b46793b2f9958e6b4cd2470dd1e44949"
server_key_version: "encryptionkey@0"
I20260620 19:14:26.387604 31976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982595231230-9837-0/minicluster-data/master-0/wal/instance:
uuid: "382f3ca60c98404686ba31891d46e66c"
format_stamp: "Formatted at 2026-06-20 19:14:26 on dist-test-slave-n3fx"
server_key: "2a57e115bb980d86ef924a43e1668959"
server_key_iv: "b46793b2f9958e6b4cd2470dd1e44949"
server_key_version: "encryptionkey@0"
I20260620 19:14:26.391399 31976 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 19:14:26.394037 31991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:26.395325 31976 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 19:14:26.395449 31976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982595231230-9837-0/minicluster-data/master-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982595231230-9837-0/minicluster-data/master-0/wal
uuid: "382f3ca60c98404686ba31891d46e66c"
format_stamp: "Formatted at 2026-06-20 19:14:26 on dist-test-slave-n3fx"
server_key: "2a57e115bb980d86ef924a43e1668959"
server_key_iv: "b46793b2f9958e6b4cd2470dd1e44949"
server_key_version: "encryptionkey@0"
I20260620 19:14:26.395555 31976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982595231230-9837-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982595231230-9837-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982595231230-9837-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:14:26.400178 31976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:14:26.403077 31976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:14:26.403257 31976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:14:26.409925 31976 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.126:42745
I20260620 19:14:26.410037 32043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.126:42745 every 8 connection(s)
I20260620 19:14:26.410888 31976 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
I20260620 19:14:26.412151 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 31976
I20260620 19:14:26.412313 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982595231230-9837-0/minicluster-data/master-0/wal/instance
I20260620 19:14:26.412609 9837 external_mini_cluster.cc:1428] Setting key 007dcb3f91b227acc5b86069cb4ca373
I20260620 19:14:26.414024 32044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:14:26.417745 32047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:26.414042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53597 (local address 127.9.155.126:42745)
0620 19:14:26.414473 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:26.414488 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:26.414523 (+ 35us) server_negotiation.cc:407] Connection header received
0620 19:14:26.415188 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:26.415205 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:26.415498 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:26.415808 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:26.416609 (+ 801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:26.416614 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:26.416626 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:14:26.416636 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:26.416689 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:14:26.416708 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:26.416844 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:26.416910 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 19:14:26.417061 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":146,"threads_started":1}
I20260620 19:14:26.419900 32044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 382f3ca60c98404686ba31891d46e66c: Bootstrap starting.
I20260620 19:14:26.420341 9837 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:14:26.422343 32044 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 382f3ca60c98404686ba31891d46e66c: Neither blocks nor log segments found. Creating new log.
I20260620 19:14:26.423134 32044 log.cc:824] T 00000000000000000000000000000000 P 382f3ca60c98404686ba31891d46e66c: Log is configured to *not* fsync() on all Append() calls
I20260620 19:14:26.423847 32047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:26.422277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44479 (local address 127.9.155.126:42745)
0620 19:14:26.422386 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:26.422390 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:26.422580 (+ 190us) server_negotiation.cc:407] Connection header received
0620 19:14:26.422696 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:26.422699 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:26.422735 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:26.422797 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:26.423495 (+ 698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:26.423498 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:26.423502 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:14:26.423512 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:26.423539 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:14:26.423550 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:26.423619 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:26.423689 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 19:14:26.423728 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 19:14:26.425272 32044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 382f3ca60c98404686ba31891d46e66c: No bootstrap required, opened a new log
I20260620 19:14:26.428797 32044 raft_consensus.cc:348] T 00000000000000000000000000000000 P 382f3ca60c98404686ba31891d46e66c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "382f3ca60c98404686ba31891d46e66c" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 42745 } }
I20260620 19:14:26.429023 32044 raft_consensus.cc:374] T 00000000000000000000000000000000 P 382f3ca60c98404686ba31891d46e66c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:14:26.429062 32044 raft_consensus.cc:729] T 00000000000000000000000000000000 P 382f3ca60c98404686ba31891d46e66c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 382f3ca60c98404686ba31891d46e66c, State: Initialized, Role: FOLLOWER
I20260620 19:14:26.429435 32044 consensus_queue.cc:260] T 00000000000000000000000000000000 P 382f3ca60c98404686ba31891d46e66c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "382f3ca60c98404686ba31891d46e66c" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 42745 } }
I20260620 19:14:26.429554 32044 raft_consensus.cc:388] T 00000000000000000000000000000000 P 382f3ca60c98404686ba31891d46e66c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:14:26.429598 32044 raft_consensus.cc:482] T 00000000000000000000000000000000 P 382f3ca60c98404686ba31891d46e66c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:14:26.429665 32044 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 382f3ca60c98404686ba31891d46e66c [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:14:26.430480 32044 raft_consensus.cc:504] T 00000000000000000000000000000000 P 382f3ca60c98404686ba31891d46e66c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "382f3ca60c98404686ba31891d46e66c" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 42745 } }
I20260620 19:14:26.430758 32044 leader_election.cc:302] T 00000000000000000000000000000000 P 382f3ca60c98404686ba31891d46e66c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 382f3ca60c98404686ba31891d46e66c; no voters:
I20260620 19:14:26.431013 32044 leader_election.cc:288] T 00000000000000000000000000000000 P 382f3ca60c98404686ba31891d46e66c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:14:26.431124 32054 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 382f3ca60c98404686ba31891d46e66c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:14:26.431339 32054 raft_consensus.cc:686] T 00000000000000000000000000000000 P 382f3ca60c98404686ba31891d46e66c [term 1 LEADER]: Becoming Leader. State: Replica: 382f3ca60c98404686ba31891d46e66c, State: Running, Role: LEADER
I20260620 19:14:26.431658 32054 consensus_queue.cc:237] T 00000000000000000000000000000000 P 382f3ca60c98404686ba31891d46e66c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "382f3ca60c98404686ba31891d46e66c" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 42745 } }
I20260620 19:14:26.432170 32044 sys_catalog.cc:564] T 00000000000000000000000000000000 P 382f3ca60c98404686ba31891d46e66c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:14:26.433456 32055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 382f3ca60c98404686ba31891d46e66c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "382f3ca60c98404686ba31891d46e66c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "382f3ca60c98404686ba31891d46e66c" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 42745 } } }
I20260620 19:14:26.433471 32056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 382f3ca60c98404686ba31891d46e66c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 382f3ca60c98404686ba31891d46e66c. Latest consensus state: current_term: 1 leader_uuid: "382f3ca60c98404686ba31891d46e66c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "382f3ca60c98404686ba31891d46e66c" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 42745 } } }
I20260620 19:14:26.433626 32055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 382f3ca60c98404686ba31891d46e66c [sys.catalog]: This master's current role is: LEADER
I20260620 19:14:26.433642 32056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 382f3ca60c98404686ba31891d46e66c [sys.catalog]: This master's current role is: LEADER
W20260620 19:14:26.436728 32069 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 382f3ca60c98404686ba31891d46e66c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:14:26.436796 32069 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:14:26.436866 32065 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:14:26.437561 32065 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:14:26.442565 32065 catalog_manager.cc:1347] Generated new cluster ID: 7272c11fe0a240afa021a35e57d39337
I20260620 19:14:26.442629 32065 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:14:26.464870 32065 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:14:26.465276 32065 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:14:26.471216 32065 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 382f3ca60c98404686ba31891d46e66c: Generated new TSK 0
I20260620 19:14:26.471954 32065 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:14:26.478852 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 31976
2026-06-20T19:14:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1311 ms)
[----------] 18 tests from AuthTokenIssuingTest (24015 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982595231230-9837-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:14:26 dist-test-slave-n3fx krb5kdc[32075](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:14:26 dist-test-slave-n3fx krb5kdc[32075](info): set up 2 sockets
Jun 20 19:14:26 dist-test-slave-n3fx krb5kdc[32075](info): commencing operation
krb5kdc: starting...
W20260620 19:14:27.428361 9837 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982595231230-9837-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982595231230-9837-0/krb5kdc/test-user.keytab.
Jun 20 19:14:27 dist-test-slave-n3fx krb5kdc[32075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982867, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:14:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:14:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.155.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.155.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.155.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab.
Entry for principal kudu/127.9.155.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab.
I20260620 19:14:27.550938 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982595231230-9837-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982595231230-9837-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982595231230-9837-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:35083
--webserver_interface=127.9.155.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab
--principal=kudu/127.9.155.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:40511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.155.126:35083,127.9.155.125:46237,127.9.155.124:39989
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:14:27.659978 32088 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 19:14:27.660337 32088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:14:27.660382 32088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:14:27.660447 32088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:14:27.664585 32088 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:14:27.664651 32088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:14:27.664676 32088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:14:27.664700 32088 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:14:27.664728 32088 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:14:27.670101 32088 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:40511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982595231230-9837-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982595231230-9837-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.155.126:35083,127.9.155.125:46237,127.9.155.124:39989
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab
--principal=kudu/127.9.155.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:35083
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.155.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982595231230-9837-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:14:27.671203 32088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:14:27.672237 32088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:14:27.680573 32093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:14:27.680660 32088 server_base.cc:1034] running on GCE node
W20260620 19:14:27.680567 32096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:14:27.680575 32094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:14:27.681468 32088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:14:27.682607 32088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:14:27.683863 32088 hybrid_clock.cc:648] HybridClock initialized: now 1781982867683843 us; error 36 us; skew 500 ppm
Jun 20 19:14:27 dist-test-slave-n3fx krb5kdc[32075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982867, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:14:27.686702 32088 init.cc:377] Logged in from keytab as kudu/127.9.155.126@KRBTEST.COM (short username kudu)
I20260620 19:14:27.687695 32088 webserver.cc:460] Webserver started at http://127.9.155.126:46029/ using document root <none> and password file <none>
I20260620 19:14:27.688292 32088 fs_manager.cc:359] Metadata directory not provided
I20260620 19:14:27.688366 32088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:14:27.688570 32088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:14:27.691030 32088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982595231230-9837-0/minicluster-data/master-0/data/instance:
uuid: "c469a9978ee04c85befc9f4c401f299c"
format_stamp: "Formatted at 2026-06-20 19:14:27 on dist-test-slave-n3fx"
server_key: "3a121d7f7b75740d1f06cfa9ccac1590"
server_key_iv: "87a45a712890c4c82d1b04dff10edd53"
server_key_version: "encryptionkey@0"
I20260620 19:14:27.691645 32088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982595231230-9837-0/minicluster-data/master-0/wal/instance:
uuid: "c469a9978ee04c85befc9f4c401f299c"
format_stamp: "Formatted at 2026-06-20 19:14:27 on dist-test-slave-n3fx"
server_key: "3a121d7f7b75740d1f06cfa9ccac1590"
server_key_iv: "87a45a712890c4c82d1b04dff10edd53"
server_key_version: "encryptionkey@0"
I20260620 19:14:27.695488 32088 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 19:14:27.698086 32103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:27.699354 32088 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 19:14:27.699489 32088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982595231230-9837-0/minicluster-data/master-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982595231230-9837-0/minicluster-data/master-0/wal
uuid: "c469a9978ee04c85befc9f4c401f299c"
format_stamp: "Formatted at 2026-06-20 19:14:27 on dist-test-slave-n3fx"
server_key: "3a121d7f7b75740d1f06cfa9ccac1590"
server_key_iv: "87a45a712890c4c82d1b04dff10edd53"
server_key_version: "encryptionkey@0"
I20260620 19:14:27.699596 32088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982595231230-9837-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982595231230-9837-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982595231230-9837-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:14:27.708786 32088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:14:27.711833 32088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:14:27.712011 32088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:14:27.718560 32088 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.126:35083
I20260620 19:14:27.718820 32155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.126:35083 every 8 connection(s)
I20260620 19:14:27.719496 32088 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
I20260620 19:14:27.722332 32156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:14:27.726157 32156 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.155.126" port: 35083 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:14:27.727043 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 32088
I20260620 19:14:27.727206 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982595231230-9837-0/minicluster-data/master-0/wal/instance
I20260620 19:14:27.727470 9837 external_mini_cluster.cc:1428] Setting key 10383755515f5e27352ce583e6863fba
WARNING: no policy specified for kudu/127.9.155.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.155.125@KRBTEST.COM" created.
Jun 20 19:14:27 dist-test-slave-n3fx krb5kdc[32075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982867, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.126@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:14:27.744943 32158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:27.728662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46062 (local address 127.9.155.126:35083)
0620 19:14:27.729093 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:27.729111 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:27.729398 (+ 287us) server_negotiation.cc:407] Connection header received
0620 19:14:27.730676 (+ 1278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:27.730681 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:27.730884 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:27.731046 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:27.732065 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:27.733032 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:27.734509 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:27.734990 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:27.735128 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:27.739873 (+ 4745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:27.739880 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:27.739893 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:14:27.739918 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:27.741902 (+ 1984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:27.742375 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:27.742381 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:27.742383 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:27.742521 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:27.742823 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:27.742827 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:27.742829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:27.743361 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:27.743506 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:27.743864 (+ 358us) server_negotiation.cc:299] Negotiation successful
0620 19:14:27.744190 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":86,"server-negotiator.queue_time_us":288,"thread_start_us":116,"threads_started":1}
I20260620 19:14:27.744940 32159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:27.728663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:35083 (local address 127.0.0.1:46062)
0620 19:14:27.729216 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:27.729271 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 19:14:27.730339 (+ 1068us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:14:27.731095 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:14:27.731101 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:14:27.731546 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:14:27.731903 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:27.731924 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:27.733168 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:27.733171 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:27.734361 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:27.734373 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:27.735167 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:27.735172 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:27.735316 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:27.736033 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:14:27.736056 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:14:27.739679 (+ 3623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:14:27.742031 (+ 2352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:27.742040 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:27.742048 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:27.742273 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:27.742600 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:27.742602 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:27.742603 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:27.742732 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:27.743480 (+ 748us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:14:27.743487 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:14:27.743616 (+ 129us) client_negotiation.cc:769] Sending connection context
0620 19:14:27.743759 (+ 143us) client_negotiation.cc:240] Negotiation successful
0620 19:14:27.744069 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"spinlock_wait_cycles":17664,"thread_start_us":149,"threads_started":1}
I20260620 19:14:27.747360 32156 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.155.125" port: 46237 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:14:27.748126 32159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:27.747832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.125:46237 (local address 127.0.0.1:44870)
0620 19:14:27.747988 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:27.748063 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.125:46237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 19:14:27.748345 32107 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.155.125:46237: connect: Connection refused (error 111)
I20260620 19:14:27.749145 32159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:27.748947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.125:46237 (local address 127.0.0.1:44872)
0620 19:14:27.749053 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:27.749095 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.125:46237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 19:14:27.749264 32156 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.155.125:46237: Network error: Client connection negotiation failed: client connection to 127.9.155.125:46237: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.155.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.125.keytab.
Entry for principal kudu/127.9.155.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.125.keytab.
I20260620 19:14:27.754514 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982595231230-9837-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982595231230-9837-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982595231230-9837-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982595231230-9837-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.125:46237
--webserver_interface=127.9.155.125
--webserver_port=0
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.125.keytab
--principal=kudu/127.9.155.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:40511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.155.126:35083,127.9.155.125:46237,127.9.155.124:39989
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:14:27.798508 32156 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.155.125" port: 46237 } attempt: 1
I20260620 19:14:27.799407 32159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:27.799071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.125:46237 (local address 127.0.0.1:44882)
0620 19:14:27.799234 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:27.799313 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.125:46237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260620 19:14:27.800189 32159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:27.799976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.125:46237 (local address 127.0.0.1:44898)
0620 19:14:27.800076 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:27.800127 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.125:46237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260620 19:14:27.800395 32156 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.155.125:46237: Network error: Client connection negotiation failed: client connection to 127.9.155.125:46237: connect: Connection refused (error 111)
W20260620 19:14:27.863463 32163 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 19:14:27.863740 32163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:14:27.863770 32163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:14:27.863808 32163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:14:27.867777 32163 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:14:27.867843 32163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:14:27.867861 32163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:14:27.867877 32163 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:14:27.867892 32163 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:14:27.868578 32156 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.155.125" port: 46237 } attempt: 2
I20260620 19:14:27.869462 32159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:27.869110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.125:46237 (local address 127.0.0.1:44914)
0620 19:14:27.869303 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:27.869374 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.125:46237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260620 19:14:27.870183 32159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:27.869973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.125:46237 (local address 127.0.0.1:44922)
0620 19:14:27.870070 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:27.870120 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.125:46237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 19:14:27.870311 32156 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.155.125:46237: Network error: Client connection negotiation failed: client connection to 127.9.155.125:46237: connect: Connection refused (error 111)
I20260620 19:14:27.873960 32163 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:40511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982595231230-9837-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982595231230-9837-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.9.155.126:35083,127.9.155.125:46237,127.9.155.124:39989
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.125.keytab
--principal=kudu/127.9.155.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.125:46237
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982595231230-9837-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.9.155.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32163
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982595231230-9837-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:14:27.875005 32163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:14:27.875974 32163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:14:27.884163 32168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:14:27.884167 32169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:14:27.884265 32163 server_base.cc:1034] running on GCE node
W20260620 19:14:27.884163 32171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:14:27.885021 32163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:14:27.886137 32163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:14:27.887307 32163 hybrid_clock.cc:648] HybridClock initialized: now 1781982867887288 us; error 32 us; skew 500 ppm
Jun 20 19:14:27 dist-test-slave-n3fx krb5kdc[32075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982867, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:14:27.890401 32163 init.cc:377] Logged in from keytab as kudu/127.9.155.125@KRBTEST.COM (short username kudu)
I20260620 19:14:27.891363 32163 webserver.cc:460] Webserver started at http://127.9.155.125:37855/ using document root <none> and password file <none>
I20260620 19:14:27.891916 32163 fs_manager.cc:359] Metadata directory not provided
I20260620 19:14:27.891983 32163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:14:27.892184 32163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:14:27.894323 32163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982595231230-9837-0/minicluster-data/master-1/data/instance:
uuid: "c9a51f649b404cdbb37cfb5abcc56074"
format_stamp: "Formatted at 2026-06-20 19:14:27 on dist-test-slave-n3fx"
server_key: "2b696c01c6b8f48b3645af138beee838"
server_key_iv: "529e443c7a8703aed4badae3110481f9"
server_key_version: "encryptionkey@0"
I20260620 19:14:27.894868 32163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982595231230-9837-0/minicluster-data/master-1/wal/instance:
uuid: "c9a51f649b404cdbb37cfb5abcc56074"
format_stamp: "Formatted at 2026-06-20 19:14:27 on dist-test-slave-n3fx"
server_key: "2b696c01c6b8f48b3645af138beee838"
server_key_iv: "529e443c7a8703aed4badae3110481f9"
server_key_version: "encryptionkey@0"
I20260620 19:14:27.898844 32163 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260620 19:14:27.901578 32178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:27.902992 32163 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 19:14:27.903131 32163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982595231230-9837-0/minicluster-data/master-1/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982595231230-9837-0/minicluster-data/master-1/wal
uuid: "c9a51f649b404cdbb37cfb5abcc56074"
format_stamp: "Formatted at 2026-06-20 19:14:27 on dist-test-slave-n3fx"
server_key: "2b696c01c6b8f48b3645af138beee838"
server_key_iv: "529e443c7a8703aed4badae3110481f9"
server_key_version: "encryptionkey@0"
I20260620 19:14:27.903244 32163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982595231230-9837-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982595231230-9837-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982595231230-9837-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:14:27.926916 32163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:14:27.930485 32163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:14:27.930686 32163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:14:27.937752 32163 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.125:46237
I20260620 19:14:27.938059 32230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.125:46237 every 8 connection(s)
I20260620 19:14:27.938786 32163 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982595231230-9837-0/minicluster-data/master-1/data/info.pb
I20260620 19:14:27.940604 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 32163
I20260620 19:14:27.940722 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982595231230-9837-0/minicluster-data/master-1/wal/instance
I20260620 19:14:27.940999 9837 external_mini_cluster.cc:1428] Setting key 0143462bec92dea11c6f8539a1c4c212
I20260620 19:14:27.941763 32231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:14:27.945647 32231 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.155.126" port: 35083 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.9.155.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.155.124@KRBTEST.COM" created.
Jun 20 19:14:27 dist-test-slave-n3fx krb5kdc[32075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982867, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.125@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:14:27.960376 32158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:27.947546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46064 (local address 127.9.155.126:35083)
0620 19:14:27.947716 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:27.947722 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:27.948586 (+ 864us) server_negotiation.cc:407] Connection header received
0620 19:14:27.949567 (+ 981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:27.949571 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:27.949624 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:27.949706 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:27.950949 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:27.951683 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:27.953370 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:27.953612 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:27.953709 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:27.956486 (+ 2777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:27.956491 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:27.956494 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:14:27.956526 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:27.958056 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:27.958619 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:27.958622 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:27.958624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:27.958704 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:27.959065 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:27.959068 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:27.959069 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:27.959263 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:27.959367 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:27.960000 (+ 633us) server_negotiation.cc:299] Negotiation successful
0620 19:14:27.960151 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 19:14:27.961247 32234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:27.947889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:35083 (local address 127.0.0.1:46064)
0620 19:14:27.948418 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:27.948476 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 19:14:27.949316 (+ 840us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:14:27.949909 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:14:27.949918 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:14:27.950294 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:14:27.950794 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:27.950809 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:27.951835 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:27.951839 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:27.953228 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:27.953239 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:27.953771 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:27.953775 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:27.953991 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:27.954584 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:14:27.954606 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:14:27.956303 (+ 1697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:14:27.958204 (+ 1901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:27.958210 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:27.958224 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:27.958500 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:27.958790 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:27.958792 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:27.958794 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:27.958987 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:27.959388 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:14:27.959394 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:14:27.959688 (+ 294us) client_negotiation.cc:769] Sending connection context
0620 19:14:27.959879 (+ 191us) client_negotiation.cc:240] Negotiation successful
0620 19:14:27.960247 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":123,"threads_started":1}
I20260620 19:14:27.961519 32156 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.155.125" port: 46237 } attempt: 3
I20260620 19:14:27.962949 32231 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.155.125" port: 46237 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.155.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.124.keytab.
Entry for principal kudu/127.9.155.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.124.keytab.
I20260620 19:14:27.966617 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982595231230-9837-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982595231230-9837-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982595231230-9837-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982595231230-9837-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.124:39989
--webserver_interface=127.9.155.124
--webserver_port=0
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.124.keytab
--principal=kudu/127.9.155.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:40511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.155.126:35083,127.9.155.125:46237,127.9.155.124:39989
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 19:14:27 dist-test-slave-n3fx krb5kdc[32075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982867, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.126@KRBTEST.COM for kudu/127.9.155.125@KRBTEST.COM
Jun 20 19:14:27 dist-test-slave-n3fx krb5kdc[32075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982867, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.125@KRBTEST.COM for kudu/127.9.155.125@KRBTEST.COM
I20260620 19:14:27.972203 32159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:27.961946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.125:46237 (local address 127.0.0.1:44924)
0620 19:14:27.962063 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:27.962077 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:14:27.962163 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:14:27.962879 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:14:27.962883 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:14:27.963069 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:14:27.963171 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:27.963178 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:27.964475 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:27.964479 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:27.965700 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:27.965711 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:27.966530 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:27.966534 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:27.966646 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:27.967148 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:14:27.967164 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:14:27.968542 (+ 1378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:14:27.970430 (+ 1888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:27.970433 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:27.970436 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:27.970633 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:27.971124 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:27.971127 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:27.971129 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:27.971212 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:27.971664 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:14:27.971667 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:14:27.971782 (+ 115us) client_negotiation.cc:769] Sending connection context
0620 19:14:27.971871 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 19:14:27.972024 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 19:14:27.972358 32236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:27.962154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44924 (local address 127.9.155.125:46237)
0620 19:14:27.962495 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:27.962503 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:27.962524 (+ 21us) server_negotiation.cc:407] Connection header received
0620 19:14:27.962561 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:27.962565 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:27.962726 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:27.962862 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:27.963325 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:27.964353 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:27.965817 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:27.966408 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:27.966546 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:27.968672 (+ 2126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:27.968682 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:27.968701 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:14:27.968721 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:27.970321 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:27.970952 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:27.970957 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:27.970958 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:27.971032 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:27.971291 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:27.971294 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:27.971296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:27.971565 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:27.971685 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:27.971975 (+ 290us) server_negotiation.cc:299] Negotiation successful
0620 19:14:27.972123 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":245,"spinlock_wait_cycles":4864,"thread_start_us":120,"threads_started":1}
I20260620 19:14:27.973220 32234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:27.963393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.125:46237 (local address 127.0.0.1:44940)
0620 19:14:27.963553 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:27.963567 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:14:27.963636 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:14:27.963995 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:14:27.963997 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:14:27.964127 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:14:27.964253 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:27.964260 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:27.965296 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:27.965299 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:27.965855 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:27.965864 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:27.966882 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:27.966885 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:27.966973 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:27.967321 (+ 348us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:14:27.967334 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:14:27.969129 (+ 1795us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:14:27.971660 (+ 2531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:27.971663 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:27.971665 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:27.971882 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:27.972157 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:27.972160 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:27.972162 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:27.972316 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:27.972730 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:14:27.972732 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:14:27.972814 (+ 82us) client_negotiation.cc:769] Sending connection context
0620 19:14:27.972913 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 19:14:27.973040 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102}
I20260620 19:14:27.973286 32237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:27.963451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44940 (local address 127.9.155.125:46237)
0620 19:14:27.963789 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:27.963797 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:27.963810 (+ 13us) server_negotiation.cc:407] Connection header received
0620 19:14:27.963841 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:27.963843 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:27.963880 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:27.963965 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:27.964344 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:27.965159 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:27.965965 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:27.966408 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:27.966539 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:27.969277 (+ 2738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:27.969280 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:27.969283 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:14:27.969305 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:27.971548 (+ 2243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:27.971981 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:27.971984 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:27.971986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:27.972073 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:27.972407 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:27.972410 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:27.972411 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:27.972609 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:27.972695 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:27.972995 (+ 300us) server_negotiation.cc:299] Negotiation successful
0620 19:14:27.973104 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":273,"spinlock_wait_cycles":3968,"thread_start_us":153,"threads_started":1}
I20260620 19:14:27.974395 32156 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.155.124" port: 39989 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:14:27.974666 32231 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.155.124" port: 39989 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:14:27.975138 32159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:27.974885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.124:39989 (local address 127.0.0.1:40290)
0620 19:14:27.975009 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:27.975065 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.124:39989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 19:14:27.975347 32234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:27.975062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.124:39989 (local address 127.0.0.1:40292)
0620 19:14:27.975192 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:27.975278 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.124:39989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 19:14:27.975520 32182 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.155.124:39989: connect: Connection refused (error 111)
I20260620 19:14:27.975785 32159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:27.975543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.124:39989 (local address 127.0.0.1:40296)
0620 19:14:27.975670 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:27.975716 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.124:39989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260620 19:14:27.975971 32156 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.155.124:39989: Network error: Client connection negotiation failed: client connection to 127.9.155.124:39989: connect: Connection refused (error 111)
I20260620 19:14:27.976454 32234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:27.976287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.124:39989 (local address 127.0.0.1:40312)
0620 19:14:27.976364 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:27.976404 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.124:39989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260620 19:14:27.976577 32231 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.155.124:39989: Network error: Client connection negotiation failed: client connection to 127.9.155.124:39989: connect: Connection refused (error 111)
I20260620 19:14:28.007143 32156 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.155.124" port: 39989 } attempt: 1
I20260620 19:14:28.008064 32159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:28.007716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.124:39989 (local address 127.0.0.1:40324)
0620 19:14:28.007921 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:28.007984 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.124:39989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 19:14:28.008955 32159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:28.008675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.124:39989 (local address 127.0.0.1:40336)
0620 19:14:28.008805 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:28.008889 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.124:39989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260620 19:14:28.009085 32156 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.155.124:39989: Network error: Client connection negotiation failed: client connection to 127.9.155.124:39989: connect: Connection refused (error 111)
I20260620 19:14:28.025853 32231 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.155.124" port: 39989 } attempt: 1
I20260620 19:14:28.026816 32234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:28.026487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.124:39989 (local address 127.0.0.1:40344)
0620 19:14:28.026656 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:28.026732 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.124:39989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260620 19:14:28.027522 32234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:28.027306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.124:39989 (local address 127.0.0.1:40352)
0620 19:14:28.027412 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:28.027459 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.124:39989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 19:14:28.027668 32231 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.155.124:39989: Network error: Client connection negotiation failed: client connection to 127.9.155.124:39989: connect: Connection refused (error 111)
W20260620 19:14:28.081866 32238 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 19:14:28.082181 32238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:14:28.082221 32238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:14:28.082283 32238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 19:14:28.084257 32156 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.155.124" port: 39989 } attempt: 2
I20260620 19:14:28.085059 32159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:28.084782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.124:39989 (local address 127.0.0.1:40356)
0620 19:14:28.084921 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:28.084982 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.124:39989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 19:14:28.085914 32159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:28.085640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.124:39989 (local address 127.0.0.1:40360)
0620 19:14:28.085786 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:28.085854 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.124:39989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260620 19:14:28.086035 32156 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.155.124:39989: Network error: Client connection negotiation failed: client connection to 127.9.155.124:39989: connect: Connection refused (error 111)
W20260620 19:14:28.086752 32238 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:14:28.086808 32238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:14:28.086838 32238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:14:28.086867 32238 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:14:28.086897 32238 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:14:28.092869 32238 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:40511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982595231230-9837-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982595231230-9837-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.9.155.126:35083,127.9.155.125:46237,127.9.155.124:39989
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.124.keytab
--principal=kudu/127.9.155.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.124:39989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982595231230-9837-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.155.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32238
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982595231230-9837-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:14:28.094019 32238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:14:28.095081 32238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 19:14:28.095830 32231 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.155.124" port: 39989 } attempt: 2
I20260620 19:14:28.096737 32234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:28.096400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.124:39989 (local address 127.0.0.1:40366)
0620 19:14:28.096586 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:28.096652 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.124:39989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260620 19:14:28.097409 32234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:28.097196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.124:39989 (local address 127.0.0.1:40372)
0620 19:14:28.097299 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:28.097347 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.124:39989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260620 19:14:28.097515 32231 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.155.124:39989: Network error: Client connection negotiation failed: client connection to 127.9.155.124:39989: connect: Connection refused (error 111)
W20260620 19:14:28.103987 32244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:14:28.103967 32247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:14:28.104066 32238 server_base.cc:1034] running on GCE node
W20260620 19:14:28.103967 32245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:14:28.104965 32238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:14:28.106340 32238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:14:28.107543 32238 hybrid_clock.cc:648] HybridClock initialized: now 1781982868107499 us; error 64 us; skew 500 ppm
Jun 20 19:14:28 dist-test-slave-n3fx krb5kdc[32075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982868, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:14:28.110646 32238 init.cc:377] Logged in from keytab as kudu/127.9.155.124@KRBTEST.COM (short username kudu)
I20260620 19:14:28.111660 32238 webserver.cc:460] Webserver started at http://127.9.155.124:38471/ using document root <none> and password file <none>
I20260620 19:14:28.112344 32238 fs_manager.cc:359] Metadata directory not provided
I20260620 19:14:28.112414 32238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:14:28.112622 32238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:14:28.114892 32238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982595231230-9837-0/minicluster-data/master-2/data/instance:
uuid: "d5c87af7bccb4f1f9606ebd7819fc8a3"
format_stamp: "Formatted at 2026-06-20 19:14:28 on dist-test-slave-n3fx"
server_key: "d5104de500caab9f309fde4c8cdb4ecf"
server_key_iv: "28058ed972e92e5d37bea430672bb256"
server_key_version: "encryptionkey@0"
I20260620 19:14:28.115533 32238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982595231230-9837-0/minicluster-data/master-2/wal/instance:
uuid: "d5c87af7bccb4f1f9606ebd7819fc8a3"
format_stamp: "Formatted at 2026-06-20 19:14:28 on dist-test-slave-n3fx"
server_key: "d5104de500caab9f309fde4c8cdb4ecf"
server_key_iv: "28058ed972e92e5d37bea430672bb256"
server_key_version: "encryptionkey@0"
I20260620 19:14:28.119570 32238 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 19:14:28.122404 32254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:28.123764 32238 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 19:14:28.123934 32238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982595231230-9837-0/minicluster-data/master-2/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982595231230-9837-0/minicluster-data/master-2/wal
uuid: "d5c87af7bccb4f1f9606ebd7819fc8a3"
format_stamp: "Formatted at 2026-06-20 19:14:28 on dist-test-slave-n3fx"
server_key: "d5104de500caab9f309fde4c8cdb4ecf"
server_key_iv: "28058ed972e92e5d37bea430672bb256"
server_key_version: "encryptionkey@0"
I20260620 19:14:28.124058 32238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982595231230-9837-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982595231230-9837-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982595231230-9837-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:14:28.147964 32238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:14:28.151168 32238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:14:28.151338 32238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:14:28.158255 32238 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.124:39989
I20260620 19:14:28.158535 32306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.124:39989 every 8 connection(s)
I20260620 19:14:28.159251 32238 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982595231230-9837-0/minicluster-data/master-2/data/info.pb
I20260620 19:14:28.162117 32307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:14:28.164009 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 32238
I20260620 19:14:28.164126 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982595231230-9837-0/minicluster-data/master-2/wal/instance
I20260620 19:14:28.164427 9837 external_mini_cluster.cc:1428] Setting key ff3a67cf2ae081b51ab5f466a6f164e5
I20260620 19:14:28.166450 32307 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.155.126" port: 35083 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 19:14:28 dist-test-slave-n3fx krb5kdc[32075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982867, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.155.124@KRBTEST.COM
Jun 20 19:14:28 dist-test-slave-n3fx krb5kdc[32075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982868, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.124@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:14:28.178650 32158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:28.169281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46078 (local address 127.9.155.126:35083)
0620 19:14:28.169418 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:28.169423 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:28.169719 (+ 296us) server_negotiation.cc:407] Connection header received
0620 19:14:28.169852 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:28.169855 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:28.169901 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:28.169965 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:28.170708 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:28.171214 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:28.172122 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:28.172395 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:28.172491 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:28.175323 (+ 2832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:28.175326 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:28.175328 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:14:28.175354 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:28.176792 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:28.177336 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:28.177340 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:28.177342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:28.177441 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:28.177676 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:28.177678 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:28.177679 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:28.177853 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:28.177928 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:28.178376 (+ 448us) server_negotiation.cc:299] Negotiation successful
0620 19:14:28.178498 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 19:14:28.179702 32311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:28.169239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:35083 (local address 127.0.0.1:46078)
0620 19:14:28.169601 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:28.169643 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 19:14:28.169763 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:14:28.170026 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:14:28.170032 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:14:28.170376 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:14:28.170507 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:28.170515 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:28.171345 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:28.171347 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:28.172000 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:28.172010 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:28.172564 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:28.172568 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:28.172675 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:28.173116 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:14:28.173138 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:14:28.175172 (+ 2034us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:14:28.176900 (+ 1728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:28.176907 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:28.176919 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:28.177251 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:28.177523 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:28.177526 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:28.177528 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:28.177603 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:28.177942 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:14:28.177949 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:14:28.178095 (+ 146us) client_negotiation.cc:769] Sending connection context
0620 19:14:28.178265 (+ 170us) client_negotiation.cc:240] Negotiation successful
0620 19:14:28.178728 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"mutex_wait_us":187,"spinlock_wait_cycles":7552,"thread_start_us":123,"threads_started":1}
I20260620 19:14:28.179702 32310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:28.166001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40380 (local address 127.9.155.124:39989)
0620 19:14:28.166456 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:28.166472 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:28.166509 (+ 37us) server_negotiation.cc:407] Connection header received
0620 19:14:28.167277 (+ 768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:28.167311 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:28.167711 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:28.168013 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:28.169103 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:28.170422 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:28.171330 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:28.171868 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:28.171987 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:28.174734 (+ 2747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:28.174742 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:28.174758 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:14:28.174785 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:28.176399 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:28.176908 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:28.176914 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:28.176916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:28.177092 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:28.177370 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:28.177373 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:28.177374 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:28.177883 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:28.178092 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:28.178265 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 19:14:28.178570 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":7,"server-negotiator.queue_time_us":308,"thread_start_us":148,"threads_started":1}
I20260620 19:14:28.181689 32307 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.155.125" port: 46237 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:14:28.185191 32156 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.155.124" port: 39989 } attempt: 3
Jun 20 19:14:28 dist-test-slave-n3fx krb5kdc[32075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982868, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.124@KRBTEST.COM for kudu/127.9.155.125@KRBTEST.COM
Jun 20 19:14:28 dist-test-slave-n3fx krb5kdc[32075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982867, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:14:28.188648 32231 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.155.124" port: 39989 } attempt: 3
Jun 20 19:14:28 dist-test-slave-n3fx krb5kdc[32075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982867, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.126@KRBTEST.COM for kudu/127.9.155.124@KRBTEST.COM
I20260620 19:14:28.192075 32237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:28.182481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44944 (local address 127.9.155.125:46237)
0620 19:14:28.182596 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:28.182600 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:28.182614 (+ 14us) server_negotiation.cc:407] Connection header received
0620 19:14:28.182658 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:28.182660 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:28.182702 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:28.182779 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:28.183235 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:28.184099 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:28.185266 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:28.185563 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:28.185679 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:28.187985 (+ 2306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:28.187989 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:28.187992 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:14:28.188023 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:28.190149 (+ 2126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:28.190597 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:28.190601 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:28.190603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:28.190690 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:28.191070 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:28.191074 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:28.191076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:28.191304 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:28.191375 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:28.191773 (+ 398us) server_negotiation.cc:299] Negotiation successful
0620 19:14:28.191908 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 19:14:28.192709 32311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:28.182228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.125:46237 (local address 127.0.0.1:44944)
0620 19:14:28.182346 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:28.182360 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:14:28.182570 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:14:28.182816 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:14:28.182819 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:14:28.182994 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:14:28.183111 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:28.183117 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:28.184255 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:28.184259 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:28.185130 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:28.185140 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:28.185680 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:28.185684 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:28.185781 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:28.186275 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:14:28.186289 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:14:28.187832 (+ 1543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:14:28.190243 (+ 2411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:28.190246 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:28.190248 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:28.190478 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:28.190789 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:28.190792 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:28.190795 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:28.190892 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:28.191458 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:14:28.191461 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:14:28.191567 (+ 106us) client_negotiation.cc:769] Sending connection context
0620 19:14:28.192393 (+ 826us) client_negotiation.cc:240] Negotiation successful
0620 19:14:28.192538 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 19:14:28.193727 32307 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.155.124" port: 39989 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:14:28.193801 32158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:28.182199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46080 (local address 127.9.155.126:35083)
0620 19:14:28.182330 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:28.182334 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:28.182368 (+ 34us) server_negotiation.cc:407] Connection header received
0620 19:14:28.182505 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:28.182508 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:28.182557 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:28.182629 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:28.183344 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:28.184083 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:28.185804 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:28.186067 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:28.186167 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:28.188612 (+ 2445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:28.188638 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:28.188641 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:14:28.188668 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:28.191434 (+ 2766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:28.191887 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:28.191890 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:28.191892 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:28.191979 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:28.192316 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:28.192319 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:28.192321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:28.192657 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:28.192914 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:28.193536 (+ 622us) server_negotiation.cc:299] Negotiation successful
0620 19:14:28.193657 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 19:14:28.194583 32159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:28.185612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.124:39989 (local address 127.0.0.1:40382)
0620 19:14:28.185872 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:28.185886 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:14:28.185974 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:14:28.186431 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:14:28.186434 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:14:28.186621 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:14:28.186714 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:28.186720 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:28.187672 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:28.187683 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:28.188438 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:28.188450 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:28.189336 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:28.189340 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:28.189420 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:28.189780 (+ 360us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:14:28.189799 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:14:28.191241 (+ 1442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:14:28.193139 (+ 1898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:28.193143 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:28.193145 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:28.193363 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:28.193664 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:28.193667 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:28.193669 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:28.193744 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:28.194162 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:14:28.194165 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:14:28.194258 (+ 93us) client_negotiation.cc:769] Sending connection context
0620 19:14:28.194330 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 19:14:28.194450 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189}
I20260620 19:14:28.195072 32310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:28.185687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40382 (local address 127.9.155.124:39989)
0620 19:14:28.185804 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:28.185808 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:28.185942 (+ 134us) server_negotiation.cc:407] Connection header received
0620 19:14:28.186276 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:28.186279 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:28.186319 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:28.186382 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:28.186814 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:28.187568 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:28.188847 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:28.189204 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:28.189613 (+ 409us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:28.191384 (+ 1771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:28.191387 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:28.191390 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:14:28.191413 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:28.193026 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:28.193485 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:28.193488 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:28.193489 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:28.193570 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:28.193845 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:28.193848 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:28.193849 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:28.194053 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:28.194743 (+ 690us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:28.194818 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 19:14:28.194936 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
Jun 20 19:14:28 dist-test-slave-n3fx krb5kdc[32075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982867, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.125@KRBTEST.COM for kudu/127.9.155.124@KRBTEST.COM
I20260620 19:14:28.198346 32156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c469a9978ee04c85befc9f4c401f299c: Bootstrap starting.
I20260620 19:14:28.200408 32234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:28.189320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.124:39989 (local address 127.0.0.1:40386)
0620 19:14:28.189422 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:28.189435 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:14:28.189503 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:14:28.190091 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:14:28.190094 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:14:28.190269 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:14:28.190369 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:28.190375 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:28.192467 (+ 2092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:28.192471 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:28.193218 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:28.193226 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:28.193774 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:28.193778 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:28.193853 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:28.194574 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:14:28.194588 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:14:28.196089 (+ 1501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:14:28.198799 (+ 2710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:28.198803 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:28.198805 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:28.199031 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:28.199329 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:28.199333 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:28.199335 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:28.199427 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:28.199841 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:14:28.199843 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:14:28.199940 (+ 97us) client_negotiation.cc:769] Sending connection context
0620 19:14:28.200059 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 19:14:28.200201 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 19:14:28.200475 32312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:28.189388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40386 (local address 127.9.155.124:39989)
0620 19:14:28.189895 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:28.189904 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:28.189920 (+ 16us) server_negotiation.cc:407] Connection header received
0620 19:14:28.189946 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:28.189948 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:28.189985 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:28.190055 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:28.190500 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:28.192356 (+ 1856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:28.193347 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:28.193657 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:28.196525 (+ 2868us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:28.196637 (+ 112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:28.196640 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:28.196643 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:14:28.196660 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:28.198699 (+ 2039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:28.199138 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:28.199142 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:28.199144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:28.199232 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:28.199523 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:28.199526 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:28.199528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:28.199743 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:28.199841 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:28.200119 (+ 278us) server_negotiation.cc:299] Negotiation successful
0620 19:14:28.200279 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":311,"threads_started":1}
Jun 20 19:14:28 dist-test-slave-n3fx krb5kdc[32075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982867, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.155.125@KRBTEST.COM
I20260620 19:14:28.201253 32156 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c469a9978ee04c85befc9f4c401f299c: Neither blocks nor log segments found. Creating new log.
I20260620 19:14:28.202162 32156 log.cc:824] T 00000000000000000000000000000000 P c469a9978ee04c85befc9f4c401f299c: Log is configured to *not* fsync() on all Append() calls
I20260620 19:14:28.203747 32231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9a51f649b404cdbb37cfb5abcc56074: Bootstrap starting.
I20260620 19:14:28.204528 32156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c469a9978ee04c85befc9f4c401f299c: No bootstrap required, opened a new log
I20260620 19:14:28.204903 32237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:28.195412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44946 (local address 127.9.155.125:46237)
0620 19:14:28.195529 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:28.195534 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:28.195549 (+ 15us) server_negotiation.cc:407] Connection header received
0620 19:14:28.195645 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:28.195648 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:28.195701 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:28.195768 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:28.196662 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:28.197378 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:28.198178 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:28.198484 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:28.198581 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:28.201221 (+ 2640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:28.201225 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:28.201227 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:14:28.201252 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:28.202887 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:28.203389 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:28.203393 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:28.203395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:28.203486 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:28.203777 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:28.203781 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:28.203782 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:28.204062 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:28.204264 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:28.204628 (+ 364us) server_negotiation.cc:299] Negotiation successful
0620 19:14:28.204743 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 19:14:28.206980 32231 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c9a51f649b404cdbb37cfb5abcc56074: Neither blocks nor log segments found. Creating new log.
Jun 20 19:14:28 dist-test-slave-n3fx krb5kdc[32075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982868, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.124@KRBTEST.COM for kudu/127.9.155.124@KRBTEST.COM
I20260620 19:14:28.208019 32231 log.cc:824] T 00000000000000000000000000000000 P c9a51f649b404cdbb37cfb5abcc56074: Log is configured to *not* fsync() on all Append() calls
I20260620 19:14:28.208653 9837 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:14:28.209779 32130 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "c469a9978ee04c85befc9f4c401f299c"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46080
I20260620 19:14:28.210574 32231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9a51f649b404cdbb37cfb5abcc56074: No bootstrap required, opened a new log
I20260620 19:14:28.210443 32156 raft_consensus.cc:348] T 00000000000000000000000000000000 P c469a9978ee04c85befc9f4c401f299c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c469a9978ee04c85befc9f4c401f299c" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 35083 } } peers { permanent_uuid: "c9a51f649b404cdbb37cfb5abcc56074" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 46237 } } peers { permanent_uuid: "d5c87af7bccb4f1f9606ebd7819fc8a3" member_type: VOTER last_known_addr { host: "127.9.155.124" port: 39989 } }
I20260620 19:14:28.210675 32156 raft_consensus.cc:729] T 00000000000000000000000000000000 P c469a9978ee04c85befc9f4c401f299c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c469a9978ee04c85befc9f4c401f299c, State: Initialized, Role: FOLLOWER
I20260620 19:14:28.211035 32156 consensus_queue.cc:260] T 00000000000000000000000000000000 P c469a9978ee04c85befc9f4c401f299c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c469a9978ee04c85befc9f4c401f299c" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 35083 } } peers { permanent_uuid: "c9a51f649b404cdbb37cfb5abcc56074" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 46237 } } peers { permanent_uuid: "d5c87af7bccb4f1f9606ebd7819fc8a3" member_type: VOTER last_known_addr { host: "127.9.155.124" port: 39989 } }
I20260620 19:14:28.211644 32316 sys_catalog.cc:455] T 00000000000000000000000000000000 P c469a9978ee04c85befc9f4c401f299c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c469a9978ee04c85befc9f4c401f299c" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 35083 } } peers { permanent_uuid: "c9a51f649b404cdbb37cfb5abcc56074" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 46237 } } peers { permanent_uuid: "d5c87af7bccb4f1f9606ebd7819fc8a3" member_type: VOTER last_known_addr { host: "127.9.155.124" port: 39989 } } }
I20260620 19:14:28.211898 32311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:28.194301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.124:39989 (local address 127.0.0.1:40396)
0620 19:14:28.194599 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:28.194612 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:14:28.195028 (+ 416us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:14:28.202799 (+ 7771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:14:28.202802 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:14:28.202971 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:14:28.203074 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:28.203081 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:28.204142 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:28.204144 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:28.204891 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:28.204902 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:28.205391 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:28.205394 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:28.205464 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:28.205975 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:14:28.205988 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:14:28.207511 (+ 1523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:14:28.210422 (+ 2911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:28.210425 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:28.210428 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:28.210672 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:28.210927 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:28.210931 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:28.210933 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:28.211012 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:28.211416 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:14:28.211420 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:14:28.211497 (+ 77us) client_negotiation.cc:769] Sending connection context
0620 19:14:28.211576 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 19:14:28.211719 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230}
I20260620 19:14:28.212333 32316 sys_catalog.cc:458] T 00000000000000000000000000000000 P c469a9978ee04c85befc9f4c401f299c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:14:28.212519 32310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:28.194341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40396 (local address 127.9.155.124:39989)
0620 19:14:28.195762 (+ 1421us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:28.195766 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:28.195784 (+ 18us) server_negotiation.cc:407] Connection header received
0620 19:14:28.195812 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:28.195814 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:28.202674 (+ 6860us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:28.203170 (+ 496us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:28.203241 (+ 71us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:28.204037 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:28.205009 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:28.205283 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:28.205375 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:28.207641 (+ 2266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:28.207645 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:28.207647 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:14:28.207688 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:28.210312 (+ 2624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:28.210762 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:28.210765 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:28.210767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:28.210836 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:28.211089 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:28.211092 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:28.211094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:28.211321 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:28.212168 (+ 847us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:28.212261 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 19:14:28.212370 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1371,"thread_start_us":133,"threads_started":1}
I20260620 19:14:28.212509 32156 sys_catalog.cc:564] T 00000000000000000000000000000000 P c469a9978ee04c85befc9f4c401f299c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:14:28.212585 32130 raft_consensus.cc:482] T 00000000000000000000000000000000 P c469a9978ee04c85befc9f4c401f299c [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:14:28.212699 32130 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c469a9978ee04c85befc9f4c401f299c [term 0 FOLLOWER]: Advancing to term 1
W20260620 19:14:28.213514 32109 tablet.cc:2369] T 00000000000000000000000000000000 P c469a9978ee04c85befc9f4c401f299c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:14:28.213965 32130 raft_consensus.cc:504] T 00000000000000000000000000000000 P c469a9978ee04c85befc9f4c401f299c [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c469a9978ee04c85befc9f4c401f299c" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 35083 } } peers { permanent_uuid: "c9a51f649b404cdbb37cfb5abcc56074" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 46237 } } peers { permanent_uuid: "d5c87af7bccb4f1f9606ebd7819fc8a3" member_type: VOTER last_known_addr { host: "127.9.155.124" port: 39989 } }
I20260620 19:14:28.214787 32130 leader_election.cc:288] T 00000000000000000000000000000000 P c469a9978ee04c85befc9f4c401f299c [CANDIDATE]: Term 1 election: Requested vote from peers c9a51f649b404cdbb37cfb5abcc56074 (127.9.155.125:46237), d5c87af7bccb4f1f9606ebd7819fc8a3 (127.9.155.124:39989)
I20260620 19:14:28.215276 32231 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9a51f649b404cdbb37cfb5abcc56074 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c469a9978ee04c85befc9f4c401f299c" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 35083 } } peers { permanent_uuid: "c9a51f649b404cdbb37cfb5abcc56074" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 46237 } } peers { permanent_uuid: "d5c87af7bccb4f1f9606ebd7819fc8a3" member_type: VOTER last_known_addr { host: "127.9.155.124" port: 39989 } }
I20260620 19:14:28.215523 32231 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9a51f649b404cdbb37cfb5abcc56074 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9a51f649b404cdbb37cfb5abcc56074, State: Initialized, Role: FOLLOWER
I20260620 19:14:28.215618 32307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5c87af7bccb4f1f9606ebd7819fc8a3: Bootstrap starting.
I20260620 19:14:28.215858 32231 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9a51f649b404cdbb37cfb5abcc56074 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c469a9978ee04c85befc9f4c401f299c" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 35083 } } peers { permanent_uuid: "c9a51f649b404cdbb37cfb5abcc56074" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 46237 } } peers { permanent_uuid: "d5c87af7bccb4f1f9606ebd7819fc8a3" member_type: VOTER last_known_addr { host: "127.9.155.124" port: 39989 } }
I20260620 19:14:28.216424 32317 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9a51f649b404cdbb37cfb5abcc56074 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c469a9978ee04c85befc9f4c401f299c" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 35083 } } peers { permanent_uuid: "c9a51f649b404cdbb37cfb5abcc56074" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 46237 } } peers { permanent_uuid: "d5c87af7bccb4f1f9606ebd7819fc8a3" member_type: VOTER last_known_addr { host: "127.9.155.124" port: 39989 } } }
I20260620 19:14:28.216641 32317 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9a51f649b404cdbb37cfb5abcc56074 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:14:28.217381 32231 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9a51f649b404cdbb37cfb5abcc56074 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:14:28.217464 32205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c469a9978ee04c85befc9f4c401f299c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c9a51f649b404cdbb37cfb5abcc56074"
I20260620 19:14:28.217711 32205 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9a51f649b404cdbb37cfb5abcc56074 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:14:28.218736 32307 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d5c87af7bccb4f1f9606ebd7819fc8a3: Neither blocks nor log segments found. Creating new log.
I20260620 19:14:28.219069 32205 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c9a51f649b404cdbb37cfb5abcc56074 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c469a9978ee04c85befc9f4c401f299c in term 1.
I20260620 19:14:28.219523 32107 leader_election.cc:302] T 00000000000000000000000000000000 P c469a9978ee04c85befc9f4c401f299c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c469a9978ee04c85befc9f4c401f299c, c9a51f649b404cdbb37cfb5abcc56074; no voters:
I20260620 19:14:28.219712 32307 log.cc:824] T 00000000000000000000000000000000 P d5c87af7bccb4f1f9606ebd7819fc8a3: Log is configured to *not* fsync() on all Append() calls
I20260620 19:14:28.219908 32316 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c469a9978ee04c85befc9f4c401f299c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:14:28.220148 32316 raft_consensus.cc:686] T 00000000000000000000000000000000 P c469a9978ee04c85befc9f4c401f299c [term 1 LEADER]: Becoming Leader. State: Replica: c469a9978ee04c85befc9f4c401f299c, State: Running, Role: LEADER
I20260620 19:14:28.220537 32316 consensus_queue.cc:237] T 00000000000000000000000000000000 P c469a9978ee04c85befc9f4c401f299c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c469a9978ee04c85befc9f4c401f299c" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 35083 } } peers { permanent_uuid: "c9a51f649b404cdbb37cfb5abcc56074" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 46237 } } peers { permanent_uuid: "d5c87af7bccb4f1f9606ebd7819fc8a3" member_type: VOTER last_known_addr { host: "127.9.155.124" port: 39989 } }
I20260620 19:14:28.222385 32316 sys_catalog.cc:455] T 00000000000000000000000000000000 P c469a9978ee04c85befc9f4c401f299c [sys.catalog]: SysCatalogTable state changed. Reason: New leader c469a9978ee04c85befc9f4c401f299c. Latest consensus state: current_term: 1 leader_uuid: "c469a9978ee04c85befc9f4c401f299c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c469a9978ee04c85befc9f4c401f299c" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 35083 } } peers { permanent_uuid: "c9a51f649b404cdbb37cfb5abcc56074" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 46237 } } peers { permanent_uuid: "d5c87af7bccb4f1f9606ebd7819fc8a3" member_type: VOTER last_known_addr { host: "127.9.155.124" port: 39989 } } }
I20260620 19:14:28.222504 32316 sys_catalog.cc:458] T 00000000000000000000000000000000 P c469a9978ee04c85befc9f4c401f299c [sys.catalog]: This master's current role is: LEADER
I20260620 19:14:28.223047 32347 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260620 19:14:28.225154 32332 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c9a51f649b404cdbb37cfb5abcc56074: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:14:28.225329 32332 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:14:28.226573 32307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5c87af7bccb4f1f9606ebd7819fc8a3: No bootstrap required, opened a new log
I20260620 19:14:28.227895 32347 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:14:28.231431 32307 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5c87af7bccb4f1f9606ebd7819fc8a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c469a9978ee04c85befc9f4c401f299c" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 35083 } } peers { permanent_uuid: "c9a51f649b404cdbb37cfb5abcc56074" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 46237 } } peers { permanent_uuid: "d5c87af7bccb4f1f9606ebd7819fc8a3" member_type: VOTER last_known_addr { host: "127.9.155.124" port: 39989 } }
I20260620 19:14:28.231735 32307 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5c87af7bccb4f1f9606ebd7819fc8a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5c87af7bccb4f1f9606ebd7819fc8a3, State: Initialized, Role: FOLLOWER
I20260620 19:14:28.221257 32163 master_runner.cc:186] Error getting master registration for 127.9.155.126:35083: OK, instance_id { permanent_uuid: "c469a9978ee04c85befc9f4c401f299c" instance_seqno: 1781982867716722 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 19:14:28.232697 32163 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 19:14:28.232127 32307 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5c87af7bccb4f1f9606ebd7819fc8a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c469a9978ee04c85befc9f4c401f299c" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 35083 } } peers { permanent_uuid: "c9a51f649b404cdbb37cfb5abcc56074" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 46237 } } peers { permanent_uuid: "d5c87af7bccb4f1f9606ebd7819fc8a3" member_type: VOTER last_known_addr { host: "127.9.155.124" port: 39989 } }
I20260620 19:14:28.234436 32307 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5c87af7bccb4f1f9606ebd7819fc8a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:14:28.234503 32281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c469a9978ee04c85befc9f4c401f299c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d5c87af7bccb4f1f9606ebd7819fc8a3"
I20260620 19:14:28.234750 32281 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5c87af7bccb4f1f9606ebd7819fc8a3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:14:28.235040 32158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:28.222959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46096 (local address 127.9.155.126:35083)
0620 19:14:28.223071 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:28.223076 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:28.225523 (+ 2447us) server_negotiation.cc:407] Connection header received
0620 19:14:28.225699 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:28.225703 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:28.225756 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:28.225809 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:28.226806 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:28.227550 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:28.228854 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:28.229180 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:28.229261 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:28.231179 (+ 1918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:28.231184 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:28.231187 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:14:28.231215 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:28.233020 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:28.233548 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:28.233552 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:28.233565 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:28.233658 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:28.233985 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:28.233988 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:28.233990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:28.234255 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:28.234359 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:28.234751 (+ 392us) server_negotiation.cc:299] Negotiation successful
0620 19:14:28.234889 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 19:14:28.236755 32281 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d5c87af7bccb4f1f9606ebd7819fc8a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c469a9978ee04c85befc9f4c401f299c in term 1.
I20260620 19:14:28.234048 32088 master_runner.cc:186] Error getting master registration for 127.9.155.124:39989: OK, instance_id { permanent_uuid: "d5c87af7bccb4f1f9606ebd7819fc8a3" instance_seqno: 1781982868156326 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 19:14:28.237447 32088 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 19:14:28.245188 32310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:28.223949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40400 (local address 127.9.155.124:39989)
0620 19:14:28.224077 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:28.224082 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:28.236753 (+ 12671us) server_negotiation.cc:407] Connection header received
0620 19:14:28.236937 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:28.236940 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:28.236991 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:28.237041 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:28.237861 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:28.238591 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:28.239624 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:28.239923 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:28.239995 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:28.241478 (+ 1483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:28.241482 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:28.241485 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:14:28.241828 (+ 343us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:28.243480 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:28.243943 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:28.243946 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:28.243949 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:28.244032 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:28.244332 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:28.244335 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:28.244336 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:28.244529 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:28.244611 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:28.244905 (+ 294us) server_negotiation.cc:299] Negotiation successful
0620 19:14:28.245022 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 19:14:28.245232 32237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:28.224329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44948 (local address 127.9.155.125:46237)
0620 19:14:28.224442 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:28.224446 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:28.228338 (+ 3892us) server_negotiation.cc:407] Connection header received
0620 19:14:28.228390 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:28.228393 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:28.228446 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:28.228522 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:28.237112 (+ 8590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:28.237913 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:28.238985 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:28.239334 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:28.239408 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:28.241257 (+ 1849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:28.241261 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:28.241265 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:14:28.241296 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:28.242984 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:28.243788 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:28.243792 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:28.243794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:28.243882 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:28.244156 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:28.244159 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:28.244161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:28.244428 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:28.244548 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:28.244901 (+ 353us) server_negotiation.cc:299] Negotiation successful
0620 19:14:28.245023 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 19:14:28.245976 32353 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5c87af7bccb4f1f9606ebd7819fc8a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c469a9978ee04c85befc9f4c401f299c" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 35083 } } peers { permanent_uuid: "c9a51f649b404cdbb37cfb5abcc56074" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 46237 } } peers { permanent_uuid: "d5c87af7bccb4f1f9606ebd7819fc8a3" member_type: VOTER last_known_addr { host: "127.9.155.124" port: 39989 } } }
I20260620 19:14:28.246101 32353 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5c87af7bccb4f1f9606ebd7819fc8a3 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:14:28.252022 32281 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d5c87af7bccb4f1f9606ebd7819fc8a3 [term 1 FOLLOWER]: Refusing update from remote peer c469a9978ee04c85befc9f4c401f299c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:14:28.252676 32334 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c469a9978ee04c85befc9f4c401f299c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5c87af7bccb4f1f9606ebd7819fc8a3" member_type: VOTER last_known_addr { host: "127.9.155.124" port: 39989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:14:28.255062 32205 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c9a51f649b404cdbb37cfb5abcc56074 [term 1 FOLLOWER]: Refusing update from remote peer c469a9978ee04c85befc9f4c401f299c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:14:28.255616 32316 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c469a9978ee04c85befc9f4c401f299c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9a51f649b404cdbb37cfb5abcc56074" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:14:28.256727 32353 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5c87af7bccb4f1f9606ebd7819fc8a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c469a9978ee04c85befc9f4c401f299c. Latest consensus state: current_term: 1 leader_uuid: "c469a9978ee04c85befc9f4c401f299c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c469a9978ee04c85befc9f4c401f299c" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 35083 } } peers { permanent_uuid: "c9a51f649b404cdbb37cfb5abcc56074" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 46237 } } peers { permanent_uuid: "d5c87af7bccb4f1f9606ebd7819fc8a3" member_type: VOTER last_known_addr { host: "127.9.155.124" port: 39989 } } }
I20260620 19:14:28.256824 32353 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5c87af7bccb4f1f9606ebd7819fc8a3 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:14:28.259521 32317 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9a51f649b404cdbb37cfb5abcc56074 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c469a9978ee04c85befc9f4c401f299c. Latest consensus state: current_term: 1 leader_uuid: "c469a9978ee04c85befc9f4c401f299c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c469a9978ee04c85befc9f4c401f299c" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 35083 } } peers { permanent_uuid: "c9a51f649b404cdbb37cfb5abcc56074" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 46237 } } peers { permanent_uuid: "d5c87af7bccb4f1f9606ebd7819fc8a3" member_type: VOTER last_known_addr { host: "127.9.155.124" port: 39989 } } }
I20260620 19:14:28.259653 32317 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9a51f649b404cdbb37cfb5abcc56074 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 19:14:28.261425 32365 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d5c87af7bccb4f1f9606ebd7819fc8a3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:14:28.261504 32365 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:14:28.261469 32316 sys_catalog.cc:455] T 00000000000000000000000000000000 P c469a9978ee04c85befc9f4c401f299c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c469a9978ee04c85befc9f4c401f299c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c469a9978ee04c85befc9f4c401f299c" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 35083 } } peers { permanent_uuid: "c9a51f649b404cdbb37cfb5abcc56074" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 46237 } } peers { permanent_uuid: "d5c87af7bccb4f1f9606ebd7819fc8a3" member_type: VOTER last_known_addr { host: "127.9.155.124" port: 39989 } } }
I20260620 19:14:28.261600 32316 sys_catalog.cc:458] T 00000000000000000000000000000000 P c469a9978ee04c85befc9f4c401f299c [sys.catalog]: This master's current role is: LEADER
I20260620 19:14:28.261844 32317 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9a51f649b404cdbb37cfb5abcc56074 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c469a9978ee04c85befc9f4c401f299c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c469a9978ee04c85befc9f4c401f299c" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 35083 } } peers { permanent_uuid: "c9a51f649b404cdbb37cfb5abcc56074" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 46237 } } peers { permanent_uuid: "d5c87af7bccb4f1f9606ebd7819fc8a3" member_type: VOTER last_known_addr { host: "127.9.155.124" port: 39989 } } }
I20260620 19:14:28.261947 32317 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9a51f649b404cdbb37cfb5abcc56074 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:14:28.262470 32369 mvcc.cc:205] Tried to move back new op lower bound from 7299001828349952000 to 7299001828235022336. Current Snapshot: MvccSnapshot[applied={T|T < 7299001828349952000}]
I20260620 19:14:28.274328 32347 catalog_manager.cc:1347] Generated new cluster ID: 1185fe67455141fb9f48ef90003c91d2
I20260620 19:14:28.274422 32347 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:14:28.275427 32316 sys_catalog.cc:455] T 00000000000000000000000000000000 P c469a9978ee04c85befc9f4c401f299c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c469a9978ee04c85befc9f4c401f299c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c469a9978ee04c85befc9f4c401f299c" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 35083 } } peers { permanent_uuid: "c9a51f649b404cdbb37cfb5abcc56074" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 46237 } } peers { permanent_uuid: "d5c87af7bccb4f1f9606ebd7819fc8a3" member_type: VOTER last_known_addr { host: "127.9.155.124" port: 39989 } } }
I20260620 19:14:28.275558 32316 sys_catalog.cc:458] T 00000000000000000000000000000000 P c469a9978ee04c85befc9f4c401f299c [sys.catalog]: This master's current role is: LEADER
I20260620 19:14:28.278311 32353 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5c87af7bccb4f1f9606ebd7819fc8a3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c469a9978ee04c85befc9f4c401f299c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c469a9978ee04c85befc9f4c401f299c" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 35083 } } peers { permanent_uuid: "c9a51f649b404cdbb37cfb5abcc56074" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 46237 } } peers { permanent_uuid: "d5c87af7bccb4f1f9606ebd7819fc8a3" member_type: VOTER last_known_addr { host: "127.9.155.124" port: 39989 } } }
I20260620 19:14:28.278432 32353 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5c87af7bccb4f1f9606ebd7819fc8a3 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:14:28.294497 32347 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:14:28.295864 32347 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:14:28.312924 32347 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c469a9978ee04c85befc9f4c401f299c: Generated new TSK 0
I20260620 19:14:28.314518 32347 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 19:14:28.346717 32158 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:14:28.343292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46104 (local address 127.9.155.126:35083)
0620 19:14:28.343424 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:28.343429 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:28.345493 (+ 2064us) server_negotiation.cc:407] Connection header received
0620 19:14:28.345698 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:28.345702 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:28.345828 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:28.345887 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:28.346559 (+ 672us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46104: BlockingRecv error: recv got EOF from 127.0.0.1:46104 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260620 19:14:28.348196 32237 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:14:28.343819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44962 (local address 127.9.155.125:46237)
0620 19:14:28.343942 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:28.343947 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:28.347225 (+ 3278us) server_negotiation.cc:407] Connection header received
0620 19:14:28.347407 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:28.347411 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:28.347460 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:28.347508 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:28.348069 (+ 561us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44962: BlockingRecv error: recv got EOF from 127.0.0.1:44962 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260620 19:14:28.349623 32310 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:14:28.344891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40416 (local address 127.9.155.124:39989)
0620 19:14:28.345031 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:28.345036 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:28.348758 (+ 3722us) server_negotiation.cc:407] Connection header received
0620 19:14:28.348926 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:28.348929 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:28.348977 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:28.349026 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:28.349503 (+ 477us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40416: BlockingRecv error: recv got EOF from 127.0.0.1:40416 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260620 19:14:28.354470 32237 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:14:28.352790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44978 (local address 127.9.155.125:46237)
0620 19:14:28.352905 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:28.352909 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:28.353375 (+ 466us) server_negotiation.cc:407] Connection header received
0620 19:14:28.353613 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:28.353616 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:28.353701 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:28.353770 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:28.354329 (+ 559us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44978: BlockingRecv error: recv got EOF from 127.0.0.1:44978 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260620 19:14:28.360178 32237 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:14:28.358418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44990 (local address 127.9.155.125:46237)
0620 19:14:28.358542 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:28.358547 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:28.359108 (+ 561us) server_negotiation.cc:407] Connection header received
0620 19:14:28.359369 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:28.359372 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:28.359430 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:28.359483 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:28.360054 (+ 571us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44990: BlockingRecv error: recv got EOF from 127.0.0.1:44990 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260620 19:14:28.366509 32237 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:14:28.364817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44992 (local address 127.9.155.125:46237)
0620 19:14:28.364939 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:28.364943 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:28.365496 (+ 553us) server_negotiation.cc:407] Connection header received
0620 19:14:28.365697 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:28.365701 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:28.365755 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:28.365820 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:28.366386 (+ 566us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44992: BlockingRecv error: recv got EOF from 127.0.0.1:44992 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260620 19:14:28.374758 32237 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:14:28.373062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44996 (local address 127.9.155.125:46237)
0620 19:14:28.373202 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:28.373207 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:28.373754 (+ 547us) server_negotiation.cc:407] Connection header received
0620 19:14:28.373979 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:28.373982 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:28.374037 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:28.374091 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:28.374643 (+ 552us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44996: BlockingRecv error: recv got EOF from 127.0.0.1:44996 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260620 19:14:28.387342 32237 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:14:28.385512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45010 (local address 127.9.155.125:46237)
0620 19:14:28.385646 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:28.385651 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:28.386294 (+ 643us) server_negotiation.cc:407] Connection header received
0620 19:14:28.386509 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:28.386512 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:28.386568 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:28.386622 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:28.387213 (+ 591us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45010: BlockingRecv error: recv got EOF from 127.0.0.1:45010 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260620 19:14:28.408128 32237 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:14:28.405888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45022 (local address 127.9.155.125:46237)
0620 19:14:28.406021 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:28.406026 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:28.406847 (+ 821us) server_negotiation.cc:407] Connection header received
0620 19:14:28.407155 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:28.407159 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:28.407216 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:28.407271 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:28.408009 (+ 738us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45022: BlockingRecv error: recv got EOF from 127.0.0.1:45022 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260620 19:14:28.443639 32237 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:14:28.442438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45030 (local address 127.9.155.125:46237)
0620 19:14:28.442560 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:28.442564 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:28.442817 (+ 253us) server_negotiation.cc:407] Connection header received
0620 19:14:28.442952 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:28.442955 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:28.443002 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:28.443070 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:28.443518 (+ 448us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45030: BlockingRecv error: recv got EOF from 127.0.0.1:45030 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260620 19:14:28.510732 32237 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:14:28.509696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45032 (local address 127.9.155.125:46237)
0620 19:14:28.509809 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:28.509813 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:28.509954 (+ 141us) server_negotiation.cc:407] Connection header received
0620 19:14:28.510041 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:28.510044 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:28.510097 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:28.510178 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:28.510611 (+ 433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45032: BlockingRecv error: recv got EOF from 127.0.0.1:45032 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260620 19:14:28.642016 32237 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:14:28.640882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45040 (local address 127.9.155.125:46237)
0620 19:14:28.641044 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:28.641048 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:28.641154 (+ 106us) server_negotiation.cc:407] Connection header received
0620 19:14:28.641322 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:28.641325 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:28.641373 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:28.641466 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:28.641917 (+ 451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45040: BlockingRecv error: recv got EOF from 127.0.0.1:45040 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260620 19:14:28.901176 32237 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:14:28.900074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45054 (local address 127.9.155.125:46237)
0620 19:14:28.900251 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:28.900255 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:28.900344 (+ 89us) server_negotiation.cc:407] Connection header received
0620 19:14:28.900487 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:28.900490 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:28.900537 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:28.900604 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:28.901056 (+ 452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45054: BlockingRecv error: recv got EOF from 127.0.0.1:45054 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 19:14:29.226745 32332 catalog_manager.cc:1259] Loaded cluster ID: 1185fe67455141fb9f48ef90003c91d2
I20260620 19:14:29.226833 32332 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c9a51f649b404cdbb37cfb5abcc56074: loading cluster ID for follower catalog manager: success
I20260620 19:14:29.228709 32332 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c9a51f649b404cdbb37cfb5abcc56074: acquiring CA information for follower catalog manager: success
I20260620 19:14:29.229866 32332 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c9a51f649b404cdbb37cfb5abcc56074: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 19:14:29.262826 32365 catalog_manager.cc:1259] Loaded cluster ID: 1185fe67455141fb9f48ef90003c91d2
I20260620 19:14:29.262920 32365 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d5c87af7bccb4f1f9606ebd7819fc8a3: loading cluster ID for follower catalog manager: success
I20260620 19:14:29.264649 32365 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d5c87af7bccb4f1f9606ebd7819fc8a3: acquiring CA information for follower catalog manager: success
I20260620 19:14:29.265839 32365 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d5c87af7bccb4f1f9606ebd7819fc8a3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 19:14:29.419375 32438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:29.415274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45060 (local address 127.9.155.125:46237)
0620 19:14:29.415545 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:29.415548 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:29.415584 (+ 36us) server_negotiation.cc:407] Connection header received
0620 19:14:29.415782 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:29.415785 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:29.415913 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:29.416014 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:14:29.416319 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:29.416898 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:29.417865 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:29.418185 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:29.418274 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:29.418552 (+ 278us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:14:29.419005 (+ 453us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:14:29.419096 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:29.419149 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 19:14:29.419207 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":94,"threads_started":1}
I20260620 19:14:29.425990 32444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:29.422399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40426 (local address 127.9.155.124:39989)
0620 19:14:29.422614 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:29.422618 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:29.422632 (+ 14us) server_negotiation.cc:407] Connection header received
0620 19:14:29.422819 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:29.422822 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:29.422943 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:29.423042 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:14:29.423205 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:29.423775 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:29.424640 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:29.424911 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:29.424994 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:29.425250 (+ 256us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:14:29.425688 (+ 438us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:14:29.425785 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:29.425817 (+ 32us) server_negotiation.cc:299] Negotiation successful
0620 19:14:29.425867 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":67,"threads_started":1}
I20260620 19:14:29.433094 32438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:29.428967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45068 (local address 127.9.155.125:46237)
0620 19:14:29.429093 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:29.429098 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:29.429590 (+ 492us) server_negotiation.cc:407] Connection header received
0620 19:14:29.429696 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:29.429698 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:29.429752 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:29.429826 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:14:29.430057 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:29.430783 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:29.431851 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:29.432170 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:29.432259 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:29.432611 (+ 352us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:14:29.432690 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:14:29.432730 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:29.432892 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 19:14:29.432950 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 19:14:29.433109 32444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:29.429460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40434 (local address 127.9.155.124:39989)
0620 19:14:29.429560 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:29.429564 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:29.430070 (+ 506us) server_negotiation.cc:407] Connection header received
0620 19:14:29.430171 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:29.430174 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:29.430221 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:29.430292 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:14:29.430515 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:29.431225 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:29.432136 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:29.432448 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:29.432540 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:29.432724 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:14:29.432791 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:14:29.432848 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:29.432937 (+ 89us) server_negotiation.cc:299] Negotiation successful
0620 19:14:29.432985 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 19:14:29.433568 32451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:29.428906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46116 (local address 127.9.155.126:35083)
0620 19:14:29.429409 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:29.429414 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:29.429427 (+ 13us) server_negotiation.cc:407] Connection header received
0620 19:14:29.429520 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:29.429523 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:29.429672 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:29.429787 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:14:29.430002 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:29.430719 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:29.431798 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:29.432150 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:29.432271 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:29.432567 (+ 296us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:14:29.432924 (+ 357us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:14:29.433342 (+ 418us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:29.433375 (+ 33us) server_negotiation.cc:299] Negotiation successful
0620 19:14:29.433433 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":88,"threads_started":1}
I20260620 19:14:29.436674 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 32088
I20260620 19:14:29.442101 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 32163
I20260620 19:14:29.447440 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 32238
2026-06-20T19:14:29Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2970 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982595231230-9837-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:14:29 dist-test-slave-n3fx krb5kdc[32455](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:14:29 dist-test-slave-n3fx krb5kdc[32455](info): set up 2 sockets
Jun 20 19:14:29 dist-test-slave-n3fx krb5kdc[32455](info): commencing operation
krb5kdc: starting...
W20260620 19:14:30.408447 9837 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982595231230-9837-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982595231230-9837-0/krb5kdc/test-user.keytab.
Jun 20 19:14:30 dist-test-slave-n3fx krb5kdc[32455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:14:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:14:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.155.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.155.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.155.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab.
Entry for principal kudu/127.9.155.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab.
I20260620 19:14:30.523278 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982595231230-9837-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982595231230-9837-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982595231230-9837-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:34667
--webserver_interface=127.9.155.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab
--principal=kudu/127.9.155.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:35039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.155.126:34667,127.9.155.125:45031,127.9.155.124:46467
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:14:30.632742 32468 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 19:14:30.633059 32468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:14:30.633102 32468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:14:30.633164 32468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:14:30.637603 32468 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:14:30.637682 32468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:14:30.637708 32468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:14:30.637729 32468 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:14:30.637759 32468 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:14:30.643312 32468 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:35039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982595231230-9837-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982595231230-9837-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.155.126:34667,127.9.155.125:45031,127.9.155.124:46467
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.126.keytab
--principal=kudu/127.9.155.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.126:34667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.155.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982595231230-9837-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:14:30.644490 32468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:14:30.645534 32468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:14:30.654047 32476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:14:30.654048 32473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:14:30.654207 32468 server_base.cc:1034] running on GCE node
W20260620 19:14:30.654078 32474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:14:30.655066 32468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:14:30.656284 32468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:14:30.657481 32468 hybrid_clock.cc:648] HybridClock initialized: now 1781982870657455 us; error 40 us; skew 500 ppm
Jun 20 19:14:30 dist-test-slave-n3fx krb5kdc[32455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982870, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:14:30.664090 32468 init.cc:377] Logged in from keytab as kudu/127.9.155.126@KRBTEST.COM (short username kudu)
I20260620 19:14:30.665133 32468 webserver.cc:460] Webserver started at http://127.9.155.126:41677/ using document root <none> and password file <none>
I20260620 19:14:30.665724 32468 fs_manager.cc:359] Metadata directory not provided
I20260620 19:14:30.665800 32468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:14:30.666015 32468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:14:30.668179 32468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982595231230-9837-0/minicluster-data/master-0/data/instance:
uuid: "e12cf6e320e74eddba8f8f7c2f7148d4"
format_stamp: "Formatted at 2026-06-20 19:14:30 on dist-test-slave-n3fx"
server_key: "fd7e573b119265eab659a27ff7185aeb"
server_key_iv: "fc0222cd16f1fdc2fa4e1434d130717b"
server_key_version: "encryptionkey@0"
I20260620 19:14:30.668840 32468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982595231230-9837-0/minicluster-data/master-0/wal/instance:
uuid: "e12cf6e320e74eddba8f8f7c2f7148d4"
format_stamp: "Formatted at 2026-06-20 19:14:30 on dist-test-slave-n3fx"
server_key: "fd7e573b119265eab659a27ff7185aeb"
server_key_iv: "fc0222cd16f1fdc2fa4e1434d130717b"
server_key_version: "encryptionkey@0"
I20260620 19:14:30.672788 32468 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 19:14:30.675324 32483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:30.676671 32468 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 19:14:30.676854 32468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982595231230-9837-0/minicluster-data/master-0/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982595231230-9837-0/minicluster-data/master-0/wal
uuid: "e12cf6e320e74eddba8f8f7c2f7148d4"
format_stamp: "Formatted at 2026-06-20 19:14:30 on dist-test-slave-n3fx"
server_key: "fd7e573b119265eab659a27ff7185aeb"
server_key_iv: "fc0222cd16f1fdc2fa4e1434d130717b"
server_key_version: "encryptionkey@0"
I20260620 19:14:30.676964 32468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982595231230-9837-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982595231230-9837-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982595231230-9837-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:14:30.695509 32468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:14:30.698805 32468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:14:30.699008 32468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:14:30.705744 32468 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.126:34667
I20260620 19:14:30.706005 32535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.126:34667 every 8 connection(s)
I20260620 19:14:30.706702 32468 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982595231230-9837-0/minicluster-data/master-0/data/info.pb
I20260620 19:14:30.709460 32536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:14:30.709798 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 32468
I20260620 19:14:30.709921 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982595231230-9837-0/minicluster-data/master-0/wal/instance
I20260620 19:14:30.710222 9837 external_mini_cluster.cc:1428] Setting key d7547d113bb84fc09c738855dd3270c1
I20260620 19:14:30.713773 32536 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.155.126" port: 34667 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.9.155.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.155.125@KRBTEST.COM" created.
Jun 20 19:14:30 dist-test-slave-n3fx krb5kdc[32455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982870, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.126@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:14:30.729786 32540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:30.717361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39874 (local address 127.9.155.126:34667)
0620 19:14:30.717681 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:30.717691 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:30.717733 (+ 42us) server_negotiation.cc:407] Connection header received
0620 19:14:30.718132 (+ 399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:30.718136 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:30.718326 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:30.718500 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:30.719336 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:30.720194 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:30.721499 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:30.721944 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:30.722075 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:30.724711 (+ 2636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:30.724719 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:30.724733 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:14:30.724761 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:30.726597 (+ 1836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:30.727100 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:30.727107 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:30.727109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:30.727259 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:30.727539 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:30.727543 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:30.727545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:30.728111 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:30.728290 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:30.728701 (+ 411us) server_negotiation.cc:299] Negotiation successful
0620 19:14:30.729057 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":149,"server-negotiator.queue_time_us":208,"thread_start_us":108,"threads_started":1}
I20260620 19:14:30.729789 32539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:30.716298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:34667 (local address 127.0.0.1:39874)
0620 19:14:30.716897 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:30.716939 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 19:14:30.717795 (+ 856us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:14:30.718509 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:14:30.718514 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:14:30.718950 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:14:30.719197 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:30.719211 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:30.720334 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:30.720336 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:30.721361 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:30.721372 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:30.722108 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:30.722113 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:30.722223 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:30.722812 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:14:30.722833 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:14:30.724548 (+ 1715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:14:30.726722 (+ 2174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:30.726730 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:30.726739 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:30.726992 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:30.727348 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:30.727350 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:30.727352 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:30.727458 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:30.728275 (+ 817us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:14:30.728283 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:14:30.728427 (+ 144us) client_negotiation.cc:769] Sending connection context
0620 19:14:30.728577 (+ 150us) client_negotiation.cc:240] Negotiation successful
0620 19:14:30.728926 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":172,"threads_started":1}
I20260620 19:14:30.732614 32536 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.155.125" port: 45031 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:14:30.733423 32539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:30.733107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.125:45031 (local address 127.0.0.1:48058)
0620 19:14:30.733246 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:30.733348 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.125:45031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260620 19:14:30.733655 32487 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.155.125:45031: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.155.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.125.keytab.
Entry for principal kudu/127.9.155.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.125.keytab.
I20260620 19:14:30.734496 32539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:30.734286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.125:45031 (local address 127.0.0.1:48060)
0620 19:14:30.734394 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:30.734441 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.125:45031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 19:14:30.734673 32536 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.155.125:45031: Network error: Client connection negotiation failed: client connection to 127.9.155.125:45031: connect: Connection refused (error 111)
I20260620 19:14:30.735280 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982595231230-9837-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982595231230-9837-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982595231230-9837-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982595231230-9837-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.125:45031
--webserver_interface=127.9.155.125
--webserver_port=0
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.125.keytab
--principal=kudu/127.9.155.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:35039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.155.126:34667,127.9.155.125:45031,127.9.155.124:46467
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:14:30.783937 32536 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.155.125" port: 45031 } attempt: 1
I20260620 19:14:30.784861 32539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:30.784538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.125:45031 (local address 127.0.0.1:48064)
0620 19:14:30.784696 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:30.784767 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.125:45031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 19:14:30.785604 32539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:30.785397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.125:45031 (local address 127.0.0.1:48078)
0620 19:14:30.785494 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:30.785544 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.125:45031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260620 19:14:30.785722 32536 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.155.125:45031: Network error: Client connection negotiation failed: client connection to 127.9.155.125:45031: connect: Connection refused (error 111)
W20260620 19:14:30.845177 32543 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 19:14:30.845486 32543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:14:30.845510 32543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:14:30.845554 32543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:14:30.849689 32543 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:14:30.849763 32543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:14:30.849782 32543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:14:30.849803 32543 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:14:30.849819 32543 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:14:30.853891 32536 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.155.125" port: 45031 } attempt: 2
I20260620 19:14:30.854740 32539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:30.854431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.125:45031 (local address 127.0.0.1:48082)
0620 19:14:30.854587 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:30.854655 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.125:45031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 19:14:30.855409 32539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:30.855203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.125:45031 (local address 127.0.0.1:48090)
0620 19:14:30.855292 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:30.855348 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.125:45031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260620 19:14:30.855522 32536 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.155.125:45031: Network error: Client connection negotiation failed: client connection to 127.9.155.125:45031: connect: Connection refused (error 111)
I20260620 19:14:30.855719 32543 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:35039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982595231230-9837-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982595231230-9837-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.9.155.126:34667,127.9.155.125:45031,127.9.155.124:46467
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.125.keytab
--principal=kudu/127.9.155.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.125:45031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982595231230-9837-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.9.155.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32543
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982595231230-9837-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:14:30.856884 32543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:14:30.857824 32543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:14:30.865926 32551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:14:30.865967 32549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:14:30.865952 32548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:14:30.866184 32543 server_base.cc:1034] running on GCE node
I20260620 19:14:30.866799 32543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:14:30.867959 32543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:14:30.869124 32543 hybrid_clock.cc:648] HybridClock initialized: now 1781982870869098 us; error 41 us; skew 500 ppm
Jun 20 19:14:30 dist-test-slave-n3fx krb5kdc[32455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982870, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:14:30.871888 32543 init.cc:377] Logged in from keytab as kudu/127.9.155.125@KRBTEST.COM (short username kudu)
I20260620 19:14:30.872828 32543 webserver.cc:460] Webserver started at http://127.9.155.125:42065/ using document root <none> and password file <none>
I20260620 19:14:30.873344 32543 fs_manager.cc:359] Metadata directory not provided
I20260620 19:14:30.873409 32543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:14:30.873585 32543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:14:30.875635 32543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982595231230-9837-0/minicluster-data/master-1/data/instance:
uuid: "2410319f235046208a95e1e84405ef27"
format_stamp: "Formatted at 2026-06-20 19:14:30 on dist-test-slave-n3fx"
server_key: "b4bfaa1d55eae7cfccb6890d08103af2"
server_key_iv: "04b112ca1360dfca74c47b98db69ed9a"
server_key_version: "encryptionkey@0"
I20260620 19:14:30.876153 32543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982595231230-9837-0/minicluster-data/master-1/wal/instance:
uuid: "2410319f235046208a95e1e84405ef27"
format_stamp: "Formatted at 2026-06-20 19:14:30 on dist-test-slave-n3fx"
server_key: "b4bfaa1d55eae7cfccb6890d08103af2"
server_key_iv: "04b112ca1360dfca74c47b98db69ed9a"
server_key_version: "encryptionkey@0"
I20260620 19:14:30.879801 32543 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:14:30.882303 32558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:30.883541 32543 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 19:14:30.883648 32543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982595231230-9837-0/minicluster-data/master-1/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982595231230-9837-0/minicluster-data/master-1/wal
uuid: "2410319f235046208a95e1e84405ef27"
format_stamp: "Formatted at 2026-06-20 19:14:30 on dist-test-slave-n3fx"
server_key: "b4bfaa1d55eae7cfccb6890d08103af2"
server_key_iv: "04b112ca1360dfca74c47b98db69ed9a"
server_key_version: "encryptionkey@0"
I20260620 19:14:30.883739 32543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982595231230-9837-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982595231230-9837-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982595231230-9837-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:14:30.897847 32543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:14:30.901134 32543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:14:30.901298 32543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:14:30.907953 32543 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.125:45031
I20260620 19:14:30.908272 32610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.125:45031 every 8 connection(s)
I20260620 19:14:30.908917 32543 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982595231230-9837-0/minicluster-data/master-1/data/info.pb
I20260620 19:14:30.911463 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 32543
I20260620 19:14:30.911592 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982595231230-9837-0/minicluster-data/master-1/wal/instance
I20260620 19:14:30.911839 32611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:14:30.911885 9837 external_mini_cluster.cc:1428] Setting key 9e9580377fc0cde5e69ca327223a10d8
I20260620 19:14:30.916164 32611 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.155.126" port: 34667 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.9.155.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.155.124@KRBTEST.COM" created.
Jun 20 19:14:30 dist-test-slave-n3fx krb5kdc[32455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982870, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.125@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:14:30.929857 32540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:30.918224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39882 (local address 127.9.155.126:34667)
0620 19:14:30.918384 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:30.918388 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:30.919467 (+ 1079us) server_negotiation.cc:407] Connection header received
0620 19:14:30.920467 (+ 1000us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:30.920470 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:30.920534 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:30.920603 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:30.921760 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:30.922300 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:30.923395 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:30.923627 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:30.923719 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:30.926331 (+ 2612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:30.926335 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:30.926337 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:14:30.926362 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:30.927692 (+ 1330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:30.928249 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:30.928253 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:30.928255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:30.928335 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:30.928685 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:30.928689 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:30.928691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:30.928888 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:30.928977 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:30.929545 (+ 568us) server_negotiation.cc:299] Negotiation successful
0620 19:14:30.929684 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 19:14:30.930680 32614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:30.918730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:34667 (local address 127.0.0.1:39882)
0620 19:14:30.919256 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:30.919353 (+ 97us) client_negotiation.cc:174] Beginning negotiation
0620 19:14:30.920243 (+ 890us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:14:30.920778 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:14:30.920785 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:14:30.921113 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:14:30.921623 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:30.921637 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:30.922422 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:30.922425 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:30.923271 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:30.923279 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:30.923766 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:30.923770 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:30.923936 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:30.924517 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:14:30.924534 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:14:30.926135 (+ 1601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:14:30.927822 (+ 1687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:30.927828 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:30.927844 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:30.928117 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:30.928434 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:30.928444 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:30.928446 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:30.928594 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:30.928990 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:14:30.928997 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:14:30.929279 (+ 282us) client_negotiation.cc:769] Sending connection context
0620 19:14:30.929465 (+ 186us) client_negotiation.cc:240] Negotiation successful
0620 19:14:30.929815 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":161,"threads_started":1}
I20260620 19:14:30.932081 32611 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.155.125" port: 45031 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.155.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.124.keytab.
Entry for principal kudu/127.9.155.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.124.keytab.
I20260620 19:14:30.935057 9837 external_mini_cluster.cc:1326] Running /tmp/dist-test-task74hYKn/build/debug/bin/kudu
/tmp/dist-test-task74hYKn/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982595231230-9837-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982595231230-9837-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982595231230-9837-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982595231230-9837-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.124:46467
--webserver_interface=127.9.155.124
--webserver_port=0
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.124.keytab
--principal=kudu/127.9.155.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.155.84:35039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.155.126:34667,127.9.155.125:45031,127.9.155.124:46467
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982595231230-9837-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982595231230-9837-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982595231230-9837-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 19:14:30 dist-test-slave-n3fx krb5kdc[32455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982870, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.125@KRBTEST.COM for kudu/127.9.155.125@KRBTEST.COM
I20260620 19:14:30.940830 32614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:30.932665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.125:45031 (local address 127.0.0.1:48098)
0620 19:14:30.932851 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:30.932866 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 19:14:30.932958 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:14:30.933441 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:14:30.933443 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:14:30.933583 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:14:30.933671 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:30.933676 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:30.934887 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:30.934890 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:30.935486 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:30.935494 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:30.936042 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:30.936045 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:30.936125 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:30.936632 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:14:30.936642 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:14:30.937760 (+ 1118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:14:30.939427 (+ 1667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:30.939430 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:30.939439 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:30.939612 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:30.939850 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:30.939853 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:30.939854 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:30.939923 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:30.940349 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:14:30.940352 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:14:30.940446 (+ 94us) client_negotiation.cc:769] Sending connection context
0620 19:14:30.940530 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 19:14:30.940656 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112}
I20260620 19:14:30.940872 32616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:30.932777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48098 (local address 127.9.155.125:45031)
0620 19:14:30.933062 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:30.933071 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:30.933090 (+ 19us) server_negotiation.cc:407] Connection header received
0620 19:14:30.933124 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:30.933129 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:30.933313 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:30.933422 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:30.933829 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:30.934777 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:30.935582 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:30.935947 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:30.936035 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:30.937873 (+ 1838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:30.937882 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:30.937894 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:14:30.937914 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:30.939335 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:30.939701 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:30.939706 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:30.939707 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:30.939776 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:30.939994 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:30.939998 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:30.940000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:30.940255 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:30.940367 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:30.940618 (+ 251us) server_negotiation.cc:299] Negotiation successful
0620 19:14:30.940743 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":106,"threads_started":1}
I20260620 19:14:30.942866 32611 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.155.124" port: 46467 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:14:30.943639 32614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:30.943382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.124:46467 (local address 127.0.0.1:38228)
0620 19:14:30.943510 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:30.943577 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.124:46467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 19:14:30.943801 32562 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.155.124:46467: connect: Connection refused (error 111)
I20260620 19:14:30.944675 32614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:30.944481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.124:46467 (local address 127.0.0.1:38244)
0620 19:14:30.944579 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:30.944625 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.124:46467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260620 19:14:30.944799 32611 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.155.124:46467: Network error: Client connection negotiation failed: client connection to 127.9.155.124:46467: connect: Connection refused (error 111)
I20260620 19:14:30.946668 32536 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.155.125" port: 45031 } attempt: 3
Jun 20 19:14:30 dist-test-slave-n3fx krb5kdc[32455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982870, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.126@KRBTEST.COM for kudu/127.9.155.125@KRBTEST.COM
I20260620 19:14:30.954485 32539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:30.947067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.125:45031 (local address 127.0.0.1:48104)
0620 19:14:30.947180 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:30.947193 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:14:30.947278 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:14:30.947542 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:14:30.947545 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:14:30.947712 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:14:30.947835 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:30.947842 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:30.948632 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:30.948636 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:30.949582 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:30.949591 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:30.950015 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:30.950019 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:30.950118 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:30.950542 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:14:30.950556 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:14:30.951648 (+ 1092us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:14:30.953145 (+ 1497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:30.953149 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:30.953151 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:30.953335 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:30.953599 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:30.953602 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:30.953604 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:30.953679 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:30.954006 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:14:30.954009 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:14:30.954101 (+ 92us) client_negotiation.cc:769] Sending connection context
0620 19:14:30.954182 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 19:14:30.954315 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 19:14:30.954485 32616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:30.947109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48104 (local address 127.9.155.125:45031)
0620 19:14:30.947245 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:30.947249 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:30.947261 (+ 12us) server_negotiation.cc:407] Connection header received
0620 19:14:30.947374 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:30.947377 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:30.947443 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:30.947504 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:30.947939 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:30.948531 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:30.949690 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:30.949919 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:30.950004 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:30.951765 (+ 1761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:30.951768 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:30.951770 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:14:30.951790 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:30.953047 (+ 1257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:30.953417 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:30.953421 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:30.953423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:30.953499 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:30.953754 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:30.953757 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:30.953759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:30.953923 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:30.954005 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:30.954239 (+ 234us) server_negotiation.cc:299] Negotiation successful
0620 19:14:30.954348 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 19:14:30.955519 32536 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.155.124" port: 46467 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:14:30.956248 32539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:30.955985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.124:46467 (local address 127.0.0.1:38252)
0620 19:14:30.956100 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:30.956154 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.124:46467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 19:14:30.956887 32539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:30.956663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.124:46467 (local address 127.0.0.1:38258)
0620 19:14:30.956785 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:30.956833 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.124:46467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260620 19:14:30.957020 32536 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.155.124:46467: Network error: Client connection negotiation failed: client connection to 127.9.155.124:46467: connect: Connection refused (error 111)
I20260620 19:14:30.988185 32536 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.155.124" port: 46467 } attempt: 1
I20260620 19:14:30.989235 32539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:30.988880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.124:46467 (local address 127.0.0.1:38262)
0620 19:14:30.989064 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:30.989139 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.124:46467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 19:14:30.990104 32539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:30.989872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.124:46467 (local address 127.0.0.1:38264)
0620 19:14:30.989978 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:30.990033 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.124:46467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 19:14:30.990268 32536 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.155.124:46467: Network error: Client connection negotiation failed: client connection to 127.9.155.124:46467: connect: Connection refused (error 111)
I20260620 19:14:30.994037 32611 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.155.124" port: 46467 } attempt: 1
I20260620 19:14:30.994830 32614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:30.994535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.124:46467 (local address 127.0.0.1:38280)
0620 19:14:30.994693 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:30.994752 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.124:46467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 19:14:30.995663 32614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:30.995439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.124:46467 (local address 127.0.0.1:38288)
0620 19:14:30.995557 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:30.995603 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.124:46467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 19:14:30.995774 32611 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.155.124:46467: Network error: Client connection negotiation failed: client connection to 127.9.155.124:46467: connect: Connection refused (error 111)
W20260620 19:14:31.048921 32617 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 19:14:31.049254 32617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:14:31.049297 32617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:14:31.049362 32617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:14:31.053592 32617 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:14:31.053655 32617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:14:31.053681 32617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:14:31.053705 32617 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:14:31.053733 32617 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:14:31.059151 32617 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.155.84:35039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982595231230-9837-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982595231230-9837-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.9.155.126:34667,127.9.155.125:45031,127.9.155.124:46467
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982595231230-9837-0/krb5kdc/kudu_127.9.155.124.keytab
--principal=kudu/127.9.155.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.155.124:46467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982595231230-9837-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.155.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982595231230-9837-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 18:22:28 UTC on ccycloud.quasar-lcwbgh.root.comops.site
build id 4457679
I20260620 19:14:31.060316 32617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:14:31.061355 32617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 19:14:31.063938 32611 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.155.124" port: 46467 } attempt: 2
I20260620 19:14:31.064765 32614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:31.064483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.124:46467 (local address 127.0.0.1:38290)
0620 19:14:31.064626 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:31.064689 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.124:46467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 19:14:31.065496 32614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:31.065241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.124:46467 (local address 127.0.0.1:38296)
0620 19:14:31.065347 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:31.065398 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.124:46467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 19:14:31.065448 32536 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.155.124" port: 46467 } attempt: 2
W20260620 19:14:31.065789 32611 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.155.124:46467: Network error: Client connection negotiation failed: client connection to 127.9.155.124:46467: connect: Connection refused (error 111)
I20260620 19:14:31.066162 32539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:31.065868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.124:46467 (local address 127.0.0.1:38304)
0620 19:14:31.066028 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:31.066084 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.124:46467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 19:14:31.066772 32539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:31.066582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.124:46467 (local address 127.0.0.1:38314)
0620 19:14:31.066682 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:31.066722 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.155.124:46467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 19:14:31.066900 32536 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.155.124:46467: Network error: Client connection negotiation failed: client connection to 127.9.155.124:46467: connect: Connection refused (error 111)
W20260620 19:14:31.070075 32626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:14:31.070065 32623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:14:31.070156 32617 server_base.cc:1034] running on GCE node
W20260620 19:14:31.070189 32624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:14:31.070909 32617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:14:31.072000 32617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:14:31.073177 32617 hybrid_clock.cc:648] HybridClock initialized: now 1781982871073152 us; error 35 us; skew 500 ppm
Jun 20 19:14:31 dist-test-slave-n3fx krb5kdc[32455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982871, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:14:31.076426 32617 init.cc:377] Logged in from keytab as kudu/127.9.155.124@KRBTEST.COM (short username kudu)
I20260620 19:14:31.077383 32617 webserver.cc:460] Webserver started at http://127.9.155.124:35095/ using document root <none> and password file <none>
I20260620 19:14:31.077943 32617 fs_manager.cc:359] Metadata directory not provided
I20260620 19:14:31.078019 32617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:14:31.078223 32617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:14:31.080574 32617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982595231230-9837-0/minicluster-data/master-2/data/instance:
uuid: "6e33e378cf6d4b4ebd9eb161f531d50c"
format_stamp: "Formatted at 2026-06-20 19:14:31 on dist-test-slave-n3fx"
server_key: "7076743bcfb22cb249546d597998f6aa"
server_key_iv: "91b89fdae24b3d4215f5e01c72ef0c6f"
server_key_version: "encryptionkey@0"
I20260620 19:14:31.081184 32617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982595231230-9837-0/minicluster-data/master-2/wal/instance:
uuid: "6e33e378cf6d4b4ebd9eb161f531d50c"
format_stamp: "Formatted at 2026-06-20 19:14:31 on dist-test-slave-n3fx"
server_key: "7076743bcfb22cb249546d597998f6aa"
server_key_iv: "91b89fdae24b3d4215f5e01c72ef0c6f"
server_key_version: "encryptionkey@0"
I20260620 19:14:31.085160 32617 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 19:14:31.087968 32633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:31.089360 32617 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 19:14:31.089491 32617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982595231230-9837-0/minicluster-data/master-2/data,/tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982595231230-9837-0/minicluster-data/master-2/wal
uuid: "6e33e378cf6d4b4ebd9eb161f531d50c"
format_stamp: "Formatted at 2026-06-20 19:14:31 on dist-test-slave-n3fx"
server_key: "7076743bcfb22cb249546d597998f6aa"
server_key_iv: "91b89fdae24b3d4215f5e01c72ef0c6f"
server_key_version: "encryptionkey@0"
I20260620 19:14:31.089605 32617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982595231230-9837-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982595231230-9837-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982595231230-9837-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:14:31.104135 32617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:14:31.107260 32617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:14:31.107448 32617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:14:31.114176 32617 rpc_server.cc:307] RPC server started. Bound to: 127.9.155.124:46467
I20260620 19:14:31.114423 32685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.155.124:46467 every 8 connection(s)
I20260620 19:14:31.115144 32617 server_base.cc:1166] Dumped server information to /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982595231230-9837-0/minicluster-data/master-2/data/info.pb
I20260620 19:14:31.118083 32686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:14:31.121413 9837 external_mini_cluster.cc:1388] Started /tmp/dist-test-task74hYKn/build/debug/bin/kudu as pid 32617
I20260620 19:14:31.121551 9837 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task74hYKn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982595231230-9837-0/minicluster-data/master-2/wal/instance
I20260620 19:14:31.121832 9837 external_mini_cluster.cc:1428] Setting key 5a5c5e11e5980698637e477353b2dc80
I20260620 19:14:31.122398 32686 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.155.126" port: 34667 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 19:14:31 dist-test-slave-n3fx krb5kdc[32455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982871, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.124@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
Jun 20 19:14:31 dist-test-slave-n3fx krb5kdc[32455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.155.124@KRBTEST.COM
I20260620 19:14:31.135004 32540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:31.124511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39898 (local address 127.9.155.126:34667)
0620 19:14:31.124668 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:31.124673 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:31.124980 (+ 307us) server_negotiation.cc:407] Connection header received
0620 19:14:31.125283 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:31.125286 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:31.125342 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:31.125471 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:31.126422 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:31.127002 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:31.128021 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:31.128285 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:31.128370 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:31.131194 (+ 2824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:31.131198 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:31.131200 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:14:31.131227 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:31.132796 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:31.133389 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:31.133392 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:31.133393 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:31.133498 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:31.133893 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:31.133896 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:31.133898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:31.134101 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:31.134188 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:31.134708 (+ 520us) server_negotiation.cc:299] Negotiation successful
0620 19:14:31.134832 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 19:14:31.135970 32690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:31.124437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.126:34667 (local address 127.0.0.1:39898)
0620 19:14:31.124822 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:31.124886 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0620 19:14:31.125084 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:14:31.125527 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:14:31.125533 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:14:31.125927 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:14:31.126245 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:31.126258 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:31.127127 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:31.127130 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:31.127897 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:31.127908 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:31.128425 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:31.128429 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:31.128623 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:31.129269 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:14:31.129285 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:14:31.131010 (+ 1725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:14:31.132913 (+ 1903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:31.132920 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:31.132932 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:31.133269 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:31.133655 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:31.133657 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:31.133659 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:31.133803 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:31.134201 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:14:31.134208 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:14:31.134480 (+ 272us) client_negotiation.cc:769] Sending connection context
0620 19:14:31.134632 (+ 152us) client_negotiation.cc:240] Negotiation successful
0620 19:14:31.135007 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":66,"spinlock_wait_cycles":896,"thread_start_us":139,"threads_started":1}
I20260620 19:14:31.135968 32689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:31.123297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38316 (local address 127.9.155.124:46467)
0620 19:14:31.123739 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:31.123756 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:31.123795 (+ 39us) server_negotiation.cc:407] Connection header received
0620 19:14:31.124648 (+ 853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:31.124675 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:31.125102 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:31.125345 (+ 243us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:31.126128 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:31.127093 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:31.127956 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:31.128408 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:31.128648 (+ 240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:31.131586 (+ 2938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:31.131592 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:31.131603 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:14:31.131625 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:31.133448 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:31.133842 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:31.133848 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:31.133850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:31.133930 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:31.134176 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:31.134179 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:31.134181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:31.134456 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:31.134622 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:31.134840 (+ 218us) server_negotiation.cc:299] Negotiation successful
0620 19:14:31.135041 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":294,"thread_start_us":126,"threads_started":1}
I20260620 19:14:31.137224 32686 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.155.125" port: 45031 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 19:14:31 dist-test-slave-n3fx krb5kdc[32455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982871, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.124@KRBTEST.COM for kudu/127.9.155.125@KRBTEST.COM
Jun 20 19:14:31 dist-test-slave-n3fx krb5kdc[32455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.155.126@KRBTEST.COM
I20260620 19:14:31.146291 32690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:31.137710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.125:45031 (local address 127.0.0.1:48106)
0620 19:14:31.137965 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:31.137980 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 19:14:31.138068 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:14:31.138300 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:14:31.138303 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:14:31.138475 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:14:31.138595 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:31.138601 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:31.139578 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:31.139582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:31.140383 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:31.140392 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:31.140934 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:31.140936 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:31.141008 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:31.141440 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:14:31.141480 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:14:31.142985 (+ 1505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:14:31.144821 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:31.144824 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:31.144827 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:31.145053 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:31.145308 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:31.145311 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:31.145312 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:31.145398 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:31.145861 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:14:31.145864 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:14:31.145944 (+ 80us) client_negotiation.cc:769] Sending connection context
0620 19:14:31.146021 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 19:14:31.146126 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172}
I20260620 19:14:31.146330 32616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:31.137825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48106 (local address 127.9.155.125:45031)
0620 19:14:31.137967 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:31.137970 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:31.138036 (+ 66us) server_negotiation.cc:407] Connection header received
0620 19:14:31.138146 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:31.138150 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:31.138206 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:31.138284 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:31.138708 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:31.139485 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:31.140487 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:31.140828 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:31.140933 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:31.143127 (+ 2194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:31.143130 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:31.143133 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:14:31.143163 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:31.144721 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:31.145139 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:31.145142 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:31.145144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:31.145228 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:31.145511 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:31.145514 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:31.145515 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:31.145708 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:31.145871 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:31.146091 (+ 220us) server_negotiation.cc:299] Negotiation successful
0620 19:14:31.146199 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 19:14:31.147097 32686 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.155.124" port: 46467 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:14:31.147302 32540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:31.138272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39902 (local address 127.9.155.126:34667)
0620 19:14:31.138380 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:31.138384 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:31.138503 (+ 119us) server_negotiation.cc:407] Connection header received
0620 19:14:31.138589 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:31.138592 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:31.138635 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:31.138708 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:31.139386 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:31.140076 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:31.141434 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:31.141736 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:31.141835 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:31.144262 (+ 2427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:31.144266 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:31.144268 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:14:31.144305 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:31.145734 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:31.146151 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:31.146154 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:31.146156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:31.146231 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:31.146528 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:31.146531 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:31.146533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:31.146726 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:31.146939 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:31.147063 (+ 124us) server_negotiation.cc:299] Negotiation successful
0620 19:14:31.147170 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Jun 20 19:14:31 dist-test-slave-n3fx krb5kdc[32455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982871, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.124@KRBTEST.COM for kudu/127.9.155.124@KRBTEST.COM
Jun 20 19:14:31 dist-test-slave-n3fx krb5kdc[32455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.155.125@KRBTEST.COM
I20260620 19:14:31.154876 32690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:31.147589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.124:46467 (local address 127.0.0.1:38320)
0620 19:14:31.147702 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:31.147716 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:14:31.147797 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:14:31.148039 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:14:31.148042 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:14:31.148192 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:14:31.148321 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:31.148327 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:31.149144 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:31.149148 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:31.149648 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:31.149655 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:31.150111 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:31.150114 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:31.150181 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:31.150565 (+ 384us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:14:31.150578 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:14:31.151812 (+ 1234us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:14:31.153549 (+ 1737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:31.153552 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:31.153554 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:31.153747 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:31.154048 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:31.154050 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:31.154051 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:31.154133 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:31.154523 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:14:31.154525 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:14:31.154597 (+ 72us) client_negotiation.cc:769] Sending connection context
0620 19:14:31.154669 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 19:14:31.154761 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 19:14:31.155021 32689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:31.147779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38320 (local address 127.9.155.124:46467)
0620 19:14:31.147870 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:31.147874 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:31.147885 (+ 11us) server_negotiation.cc:407] Connection header received
0620 19:14:31.147916 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:31.147919 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:31.147958 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:31.148021 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:31.148430 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:31.149057 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:31.149739 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:31.150029 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:31.150113 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:31.151924 (+ 1811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:31.151927 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:31.151929 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:14:31.151947 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:31.153434 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:31.153883 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:31.153886 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:31.153887 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:31.153961 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:31.154226 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:31.154229 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:31.154231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:31.154437 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:31.154539 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:31.154734 (+ 195us) server_negotiation.cc:299] Negotiation successful
0620 19:14:31.154851 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 19:14:31.156388 32616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:31.148449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48108 (local address 127.9.155.125:45031)
0620 19:14:31.148559 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:31.148563 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:31.148574 (+ 11us) server_negotiation.cc:407] Connection header received
0620 19:14:31.148649 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:31.148651 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:31.148689 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:31.148757 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:31.149236 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:31.149703 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:31.150385 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:31.150650 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:31.150751 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:31.153144 (+ 2393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:31.153146 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:31.153148 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:14:31.153168 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:31.154652 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:31.155093 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:31.155096 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:31.155098 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:31.155176 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:31.155445 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:31.155448 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:31.155450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:31.155649 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:31.155822 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:31.156053 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 19:14:31.156141 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 19:14:31.156941 32611 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.155.124" port: 46467 } attempt: 3
I20260620 19:14:31.158789 32686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e33e378cf6d4b4ebd9eb161f531d50c: Bootstrap starting.
I20260620 19:14:31.159575 9837 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:14:31.161442 32686 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e33e378cf6d4b4ebd9eb161f531d50c: Neither blocks nor log segments found. Creating new log.
I20260620 19:14:31.160629 32510 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "e12cf6e320e74eddba8f8f7c2f7148d4"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39902
I20260620 19:14:31.162256 32686 log.cc:824] T 00000000000000000000000000000000 P 6e33e378cf6d4b4ebd9eb161f531d50c: Log is configured to *not* fsync() on all Append() calls
I20260620 19:14:31.164640 32686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e33e378cf6d4b4ebd9eb161f531d50c: No bootstrap required, opened a new log
Jun 20 19:14:31 dist-test-slave-n3fx krb5kdc[32455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982870, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.125@KRBTEST.COM for kudu/127.9.155.124@KRBTEST.COM
I20260620 19:14:31.166055 32536 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.155.124" port: 46467 } attempt: 3
I20260620 19:14:31.168768 32614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:31.157286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.124:46467 (local address 127.0.0.1:38326)
0620 19:14:31.160340 (+ 3054us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:31.160354 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:14:31.160432 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:14:31.160684 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:14:31.160687 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:14:31.160876 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:14:31.160974 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:31.160980 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:31.161942 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:31.161945 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:31.162496 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:31.162503 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:31.163013 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:31.163017 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:31.163110 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:31.163619 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:14:31.163633 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:14:31.165149 (+ 1516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:14:31.167147 (+ 1998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:31.167151 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:31.167153 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:31.167350 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:31.167641 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:31.167645 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:31.167647 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:31.167758 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:31.168244 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:14:31.168247 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:14:31.168354 (+ 107us) client_negotiation.cc:769] Sending connection context
0620 19:14:31.168451 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 19:14:31.168602 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2976}
I20260620 19:14:31.168798 32686 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e33e378cf6d4b4ebd9eb161f531d50c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e12cf6e320e74eddba8f8f7c2f7148d4" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 34667 } } peers { permanent_uuid: "2410319f235046208a95e1e84405ef27" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 45031 } } peers { permanent_uuid: "6e33e378cf6d4b4ebd9eb161f531d50c" member_type: VOTER last_known_addr { host: "127.9.155.124" port: 46467 } }
I20260620 19:14:31.169032 32689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:31.157514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38326 (local address 127.9.155.124:46467)
0620 19:14:31.157608 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:31.157611 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:31.160397 (+ 2786us) server_negotiation.cc:407] Connection header received
0620 19:14:31.160531 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:31.160534 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:31.160576 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:31.160646 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:31.161083 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:31.161796 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:31.162601 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:31.162893 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:31.162996 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:31.165304 (+ 2308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:31.165309 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:31.165312 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:14:31.165340 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:31.167027 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:31.167445 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:31.167448 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:31.167450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:31.167536 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:31.167871 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:31.167874 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:31.167876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:31.168128 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:31.168236 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:31.168581 (+ 345us) server_negotiation.cc:299] Negotiation successful
0620 19:14:31.168756 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":36}
I20260620 19:14:31.169090 32686 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e33e378cf6d4b4ebd9eb161f531d50c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e33e378cf6d4b4ebd9eb161f531d50c, State: Initialized, Role: FOLLOWER
I20260620 19:14:31.169449 32686 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e33e378cf6d4b4ebd9eb161f531d50c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e12cf6e320e74eddba8f8f7c2f7148d4" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 34667 } } peers { permanent_uuid: "2410319f235046208a95e1e84405ef27" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 45031 } } peers { permanent_uuid: "6e33e378cf6d4b4ebd9eb161f531d50c" member_type: VOTER last_known_addr { host: "127.9.155.124" port: 46467 } }
I20260620 19:14:31.169929 32693 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e33e378cf6d4b4ebd9eb161f531d50c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e12cf6e320e74eddba8f8f7c2f7148d4" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 34667 } } peers { permanent_uuid: "2410319f235046208a95e1e84405ef27" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 45031 } } peers { permanent_uuid: "6e33e378cf6d4b4ebd9eb161f531d50c" member_type: VOTER last_known_addr { host: "127.9.155.124" port: 46467 } } }
I20260620 19:14:31.170054 32693 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e33e378cf6d4b4ebd9eb161f531d50c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:14:31.170850 32686 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e33e378cf6d4b4ebd9eb161f531d50c [sys.catalog]: configured and running, proceeding with master startup.
Jun 20 19:14:31 dist-test-slave-n3fx krb5kdc[32455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982870, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.155.126@KRBTEST.COM for kudu/127.9.155.124@KRBTEST.COM
I20260620 19:14:31.172688 32611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2410319f235046208a95e1e84405ef27: Bootstrap starting.
W20260620 19:14:31.175635 32704 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6e33e378cf6d4b4ebd9eb161f531d50c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:14:31.175714 32704 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:14:31.176396 32611 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2410319f235046208a95e1e84405ef27: Neither blocks nor log segments found. Creating new log.
I20260620 19:14:31.177270 32611 log.cc:824] T 00000000000000000000000000000000 P 2410319f235046208a95e1e84405ef27: Log is configured to *not* fsync() on all Append() calls
I20260620 19:14:31.174350 32617 master_runner.cc:186] Error getting master registration for 127.9.155.126:34667: OK, instance_id { permanent_uuid: "e12cf6e320e74eddba8f8f7c2f7148d4" instance_seqno: 1781982870703834 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 19:14:31.177660 32617 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 19:14:31.179682 32611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2410319f235046208a95e1e84405ef27: No bootstrap required, opened a new log
I20260620 19:14:31.180392 32539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:31.166501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.155.124:46467 (local address 127.0.0.1:38338)
0620 19:14:31.166691 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:31.166706 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 19:14:31.166788 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:14:31.167982 (+ 1194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:14:31.167986 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:14:31.168178 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:14:31.168377 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:31.168385 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:31.169413 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:31.169417 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:31.170192 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:31.170203 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:31.170755 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:31.170759 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:31.170842 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:31.171387 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:14:31.171403 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:14:31.172848 (+ 1445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:14:31.178611 (+ 5763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:31.178615 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:31.178618 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:31.178903 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:31.179237 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:31.179240 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:31.179242 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:31.179345 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:31.179784 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:14:31.179787 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:14:31.179905 (+ 118us) client_negotiation.cc:769] Sending connection context
0620 19:14:31.180014 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 19:14:31.180181 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111}
I20260620 19:14:31.180436 32692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:31.166983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38338 (local address 127.9.155.124:46467)
0620 19:14:31.167674 (+ 691us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:31.167684 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:31.167696 (+ 12us) server_negotiation.cc:407] Connection header received
0620 19:14:31.167726 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:31.167729 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:31.167787 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:31.167922 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:31.168509 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:31.169305 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:31.170307 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:31.170624 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:31.170723 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:31.173017 (+ 2294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:31.173022 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:31.173024 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:14:31.173047 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:31.178470 (+ 5423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:31.179005 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:31.179008 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:31.179010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:31.179134 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:31.179454 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:31.179457 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:31.179458 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:31.179684 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:31.179796 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:31.180088 (+ 292us) server_negotiation.cc:299] Negotiation successful
0620 19:14:31.180254 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":620,"thread_start_us":118,"threads_started":1}
I20260620 19:14:31.183691 32536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e12cf6e320e74eddba8f8f7c2f7148d4: Bootstrap starting.
I20260620 19:14:31.183607 32611 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2410319f235046208a95e1e84405ef27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e12cf6e320e74eddba8f8f7c2f7148d4" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 34667 } } peers { permanent_uuid: "2410319f235046208a95e1e84405ef27" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 45031 } } peers { permanent_uuid: "6e33e378cf6d4b4ebd9eb161f531d50c" member_type: VOTER last_known_addr { host: "127.9.155.124" port: 46467 } }
I20260620 19:14:31.183804 32611 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2410319f235046208a95e1e84405ef27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2410319f235046208a95e1e84405ef27, State: Initialized, Role: FOLLOWER
I20260620 19:14:31.184099 32611 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2410319f235046208a95e1e84405ef27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e12cf6e320e74eddba8f8f7c2f7148d4" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 34667 } } peers { permanent_uuid: "2410319f235046208a95e1e84405ef27" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 45031 } } peers { permanent_uuid: "6e33e378cf6d4b4ebd9eb161f531d50c" member_type: VOTER last_known_addr { host: "127.9.155.124" port: 46467 } }
I20260620 19:14:31.184566 32707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2410319f235046208a95e1e84405ef27 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e12cf6e320e74eddba8f8f7c2f7148d4" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 34667 } } peers { permanent_uuid: "2410319f235046208a95e1e84405ef27" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 45031 } } peers { permanent_uuid: "6e33e378cf6d4b4ebd9eb161f531d50c" member_type: VOTER last_known_addr { host: "127.9.155.124" port: 46467 } } }
I20260620 19:14:31.184765 32707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2410319f235046208a95e1e84405ef27 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:14:31.185362 32611 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2410319f235046208a95e1e84405ef27 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:14:31.186441 32536 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e12cf6e320e74eddba8f8f7c2f7148d4: Neither blocks nor log segments found. Creating new log.
I20260620 19:14:31.187639 32536 log.cc:824] T 00000000000000000000000000000000 P e12cf6e320e74eddba8f8f7c2f7148d4: Log is configured to *not* fsync() on all Append() calls
I20260620 19:14:31.190290 32536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e12cf6e320e74eddba8f8f7c2f7148d4: No bootstrap required, opened a new log
W20260620 19:14:31.190925 32718 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2410319f235046208a95e1e84405ef27: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:14:31.191041 32718 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:14:31.188680 32543 master_runner.cc:186] Error getting master registration for 127.9.155.126:34667: OK, instance_id { permanent_uuid: "e12cf6e320e74eddba8f8f7c2f7148d4" instance_seqno: 1781982870703834 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 19:14:31.191721 32543 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 19:14:31.193439 32536 raft_consensus.cc:348] T 00000000000000000000000000000000 P e12cf6e320e74eddba8f8f7c2f7148d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e12cf6e320e74eddba8f8f7c2f7148d4" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 34667 } } peers { permanent_uuid: "2410319f235046208a95e1e84405ef27" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 45031 } } peers { permanent_uuid: "6e33e378cf6d4b4ebd9eb161f531d50c" member_type: VOTER last_known_addr { host: "127.9.155.124" port: 46467 } }
I20260620 19:14:31.193651 32536 raft_consensus.cc:729] T 00000000000000000000000000000000 P e12cf6e320e74eddba8f8f7c2f7148d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e12cf6e320e74eddba8f8f7c2f7148d4, State: Initialized, Role: FOLLOWER
I20260620 19:14:31.193965 32536 consensus_queue.cc:260] T 00000000000000000000000000000000 P e12cf6e320e74eddba8f8f7c2f7148d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e12cf6e320e74eddba8f8f7c2f7148d4" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 34667 } } peers { permanent_uuid: "2410319f235046208a95e1e84405ef27" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 45031 } } peers { permanent_uuid: "6e33e378cf6d4b4ebd9eb161f531d50c" member_type: VOTER last_known_addr { host: "127.9.155.124" port: 46467 } }
I20260620 19:14:31.194595 32721 sys_catalog.cc:455] T 00000000000000000000000000000000 P e12cf6e320e74eddba8f8f7c2f7148d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e12cf6e320e74eddba8f8f7c2f7148d4" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 34667 } } peers { permanent_uuid: "2410319f235046208a95e1e84405ef27" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 45031 } } peers { permanent_uuid: "6e33e378cf6d4b4ebd9eb161f531d50c" member_type: VOTER last_known_addr { host: "127.9.155.124" port: 46467 } } }
I20260620 19:14:31.194878 32721 sys_catalog.cc:458] T 00000000000000000000000000000000 P e12cf6e320e74eddba8f8f7c2f7148d4 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:14:31.195312 32536 sys_catalog.cc:564] T 00000000000000000000000000000000 P e12cf6e320e74eddba8f8f7c2f7148d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:14:31.195369 32510 raft_consensus.cc:482] T 00000000000000000000000000000000 P e12cf6e320e74eddba8f8f7c2f7148d4 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:14:31.195478 32510 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e12cf6e320e74eddba8f8f7c2f7148d4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:14:31.196810 32510 raft_consensus.cc:504] T 00000000000000000000000000000000 P e12cf6e320e74eddba8f8f7c2f7148d4 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e12cf6e320e74eddba8f8f7c2f7148d4" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 34667 } } peers { permanent_uuid: "2410319f235046208a95e1e84405ef27" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 45031 } } peers { permanent_uuid: "6e33e378cf6d4b4ebd9eb161f531d50c" member_type: VOTER last_known_addr { host: "127.9.155.124" port: 46467 } }
I20260620 19:14:31.197768 32510 leader_election.cc:288] T 00000000000000000000000000000000 P e12cf6e320e74eddba8f8f7c2f7148d4 [CANDIDATE]: Term 1 election: Requested vote from peers 2410319f235046208a95e1e84405ef27 (127.9.155.125:45031), 6e33e378cf6d4b4ebd9eb161f531d50c (127.9.155.124:46467)
I20260620 19:14:31.198869 32660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e12cf6e320e74eddba8f8f7c2f7148d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "6e33e378cf6d4b4ebd9eb161f531d50c"
I20260620 19:14:31.199132 32660 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e33e378cf6d4b4ebd9eb161f531d50c [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:14:31.199412 32585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e12cf6e320e74eddba8f8f7c2f7148d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2410319f235046208a95e1e84405ef27"
I20260620 19:14:31.199617 32585 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2410319f235046208a95e1e84405ef27 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:14:31.200537 32660 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6e33e378cf6d4b4ebd9eb161f531d50c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e12cf6e320e74eddba8f8f7c2f7148d4 in term 1.
W20260620 19:14:31.200630 32489 tablet.cc:2369] T 00000000000000000000000000000000 P e12cf6e320e74eddba8f8f7c2f7148d4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:14:31.200903 32585 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2410319f235046208a95e1e84405ef27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e12cf6e320e74eddba8f8f7c2f7148d4 in term 1.
I20260620 19:14:31.203109 32487 leader_election.cc:302] T 00000000000000000000000000000000 P e12cf6e320e74eddba8f8f7c2f7148d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e33e378cf6d4b4ebd9eb161f531d50c, e12cf6e320e74eddba8f8f7c2f7148d4; no voters:
I20260620 19:14:31.203365 32721 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e12cf6e320e74eddba8f8f7c2f7148d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:14:31.203629 32721 raft_consensus.cc:686] T 00000000000000000000000000000000 P e12cf6e320e74eddba8f8f7c2f7148d4 [term 1 LEADER]: Becoming Leader. State: Replica: e12cf6e320e74eddba8f8f7c2f7148d4, State: Running, Role: LEADER
I20260620 19:14:31.204016 32721 consensus_queue.cc:237] T 00000000000000000000000000000000 P e12cf6e320e74eddba8f8f7c2f7148d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e12cf6e320e74eddba8f8f7c2f7148d4" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 34667 } } peers { permanent_uuid: "2410319f235046208a95e1e84405ef27" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 45031 } } peers { permanent_uuid: "6e33e378cf6d4b4ebd9eb161f531d50c" member_type: VOTER last_known_addr { host: "127.9.155.124" port: 46467 } }
I20260620 19:14:31.205874 32721 sys_catalog.cc:455] T 00000000000000000000000000000000 P e12cf6e320e74eddba8f8f7c2f7148d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e12cf6e320e74eddba8f8f7c2f7148d4. Latest consensus state: current_term: 1 leader_uuid: "e12cf6e320e74eddba8f8f7c2f7148d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e12cf6e320e74eddba8f8f7c2f7148d4" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 34667 } } peers { permanent_uuid: "2410319f235046208a95e1e84405ef27" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 45031 } } peers { permanent_uuid: "6e33e378cf6d4b4ebd9eb161f531d50c" member_type: VOTER last_known_addr { host: "127.9.155.124" port: 46467 } } }
I20260620 19:14:31.206048 32721 sys_catalog.cc:458] T 00000000000000000000000000000000 P e12cf6e320e74eddba8f8f7c2f7148d4 [sys.catalog]: This master's current role is: LEADER
I20260620 19:14:31.209146 32468 master_runner.cc:263] No leader master found from master e12cf6e320e74eddba8f8f7c2f7148d4
I20260620 19:14:31.209211 32468 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 19:14:31.211040 32732 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e12cf6e320e74eddba8f8f7c2f7148d4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:14:31.211122 32732 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:14:31.219337 32740 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:14:31.220486 32740 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:14:31.230573 32692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:31.207868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38354 (local address 127.9.155.124:46467)
0620 19:14:31.207976 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:31.207980 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:31.212472 (+ 4492us) server_negotiation.cc:407] Connection header received
0620 19:14:31.216458 (+ 3986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:31.216462 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:31.216516 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:31.216568 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:31.218466 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:31.219264 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:31.222350 (+ 3086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:31.222713 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:31.222786 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:31.224544 (+ 1758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:31.224548 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:31.224551 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:14:31.224580 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:31.226200 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:31.227008 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:31.227012 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:31.227014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:31.227105 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:31.229576 (+ 2471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:31.229580 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:31.229582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:31.229828 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:31.229940 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:31.230268 (+ 328us) server_negotiation.cc:299] Negotiation successful
0620 19:14:31.230403 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 19:14:31.235527 32616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:31.207248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48122 (local address 127.9.155.125:45031)
0620 19:14:31.207354 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:31.207358 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:31.211722 (+ 4364us) server_negotiation.cc:407] Connection header received
0620 19:14:31.212057 (+ 335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:31.212061 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:31.212111 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:31.212160 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:31.216184 (+ 4024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:31.216989 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:31.221426 (+ 4437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:31.221778 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:31.221867 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:31.231005 (+ 9138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:31.231009 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:31.231012 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:14:31.231044 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:31.233369 (+ 2325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:31.233850 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:31.233854 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:31.233856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:31.233940 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:31.234200 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:31.234203 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:31.234205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:31.234421 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:31.234525 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:31.235250 (+ 725us) server_negotiation.cc:299] Negotiation successful
0620 19:14:31.235369 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 19:14:31.238216 32540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:31.202744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39918 (local address 127.9.155.126:34667)
0620 19:14:31.202861 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:31.202865 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:31.211542 (+ 8677us) server_negotiation.cc:407] Connection header received
0620 19:14:31.212324 (+ 782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:31.212328 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:31.212371 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:31.212416 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:31.214185 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:31.214994 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:31.220446 (+ 5452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:31.220791 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:31.220875 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:31.227426 (+ 6551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:31.227430 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:31.227433 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:14:31.227465 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:31.229090 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:31.236805 (+ 7715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:31.236809 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:31.236811 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:31.236929 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:31.237274 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:31.237278 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:31.237279 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:31.237490 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:31.237596 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:31.237921 (+ 325us) server_negotiation.cc:299] Negotiation successful
0620 19:14:31.238061 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 19:14:31.239653 32660 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6e33e378cf6d4b4ebd9eb161f531d50c [term 1 FOLLOWER]: Refusing update from remote peer e12cf6e320e74eddba8f8f7c2f7148d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:14:31.240618 32738 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e12cf6e320e74eddba8f8f7c2f7148d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e33e378cf6d4b4ebd9eb161f531d50c" member_type: VOTER last_known_addr { host: "127.9.155.124" port: 46467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:14:31.245157 32693 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e33e378cf6d4b4ebd9eb161f531d50c [sys.catalog]: SysCatalogTable state changed. Reason: New leader e12cf6e320e74eddba8f8f7c2f7148d4. Latest consensus state: current_term: 1 leader_uuid: "e12cf6e320e74eddba8f8f7c2f7148d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e12cf6e320e74eddba8f8f7c2f7148d4" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 34667 } } peers { permanent_uuid: "2410319f235046208a95e1e84405ef27" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 45031 } } peers { permanent_uuid: "6e33e378cf6d4b4ebd9eb161f531d50c" member_type: VOTER last_known_addr { host: "127.9.155.124" port: 46467 } } }
I20260620 19:14:31.245321 32693 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e33e378cf6d4b4ebd9eb161f531d50c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:14:31.248728 32585 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2410319f235046208a95e1e84405ef27 [term 1 FOLLOWER]: Refusing update from remote peer e12cf6e320e74eddba8f8f7c2f7148d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:14:31.249404 32738 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e12cf6e320e74eddba8f8f7c2f7148d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2410319f235046208a95e1e84405ef27" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 45031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:14:31.253300 32707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2410319f235046208a95e1e84405ef27 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e12cf6e320e74eddba8f8f7c2f7148d4. Latest consensus state: current_term: 1 leader_uuid: "e12cf6e320e74eddba8f8f7c2f7148d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e12cf6e320e74eddba8f8f7c2f7148d4" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 34667 } } peers { permanent_uuid: "2410319f235046208a95e1e84405ef27" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 45031 } } peers { permanent_uuid: "6e33e378cf6d4b4ebd9eb161f531d50c" member_type: VOTER last_known_addr { host: "127.9.155.124" port: 46467 } } }
I20260620 19:14:31.253434 32707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2410319f235046208a95e1e84405ef27 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:14:31.257021 32721 sys_catalog.cc:455] T 00000000000000000000000000000000 P e12cf6e320e74eddba8f8f7c2f7148d4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e12cf6e320e74eddba8f8f7c2f7148d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e12cf6e320e74eddba8f8f7c2f7148d4" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 34667 } } peers { permanent_uuid: "2410319f235046208a95e1e84405ef27" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 45031 } } peers { permanent_uuid: "6e33e378cf6d4b4ebd9eb161f531d50c" member_type: VOTER last_known_addr { host: "127.9.155.124" port: 46467 } } }
I20260620 19:14:31.257156 32721 sys_catalog.cc:458] T 00000000000000000000000000000000 P e12cf6e320e74eddba8f8f7c2f7148d4 [sys.catalog]: This master's current role is: LEADER
I20260620 19:14:31.257257 32744 mvcc.cc:205] Tried to move back new op lower bound from 7299001840581218304 to 7299001840455475200. Current Snapshot: MvccSnapshot[applied={T|T < 7299001840581218304}]
I20260620 19:14:31.257797 32745 mvcc.cc:205] Tried to move back new op lower bound from 7299001840581218304 to 7299001840455475200. Current Snapshot: MvccSnapshot[applied={T|T < 7299001840581218304}]
I20260620 19:14:31.257962 32693 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e33e378cf6d4b4ebd9eb161f531d50c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e12cf6e320e74eddba8f8f7c2f7148d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e12cf6e320e74eddba8f8f7c2f7148d4" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 34667 } } peers { permanent_uuid: "2410319f235046208a95e1e84405ef27" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 45031 } } peers { permanent_uuid: "6e33e378cf6d4b4ebd9eb161f531d50c" member_type: VOTER last_known_addr { host: "127.9.155.124" port: 46467 } } }
I20260620 19:14:31.258066 32693 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e33e378cf6d4b4ebd9eb161f531d50c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:14:31.266106 32738 sys_catalog.cc:455] T 00000000000000000000000000000000 P e12cf6e320e74eddba8f8f7c2f7148d4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e12cf6e320e74eddba8f8f7c2f7148d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e12cf6e320e74eddba8f8f7c2f7148d4" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 34667 } } peers { permanent_uuid: "2410319f235046208a95e1e84405ef27" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 45031 } } peers { permanent_uuid: "6e33e378cf6d4b4ebd9eb161f531d50c" member_type: VOTER last_known_addr { host: "127.9.155.124" port: 46467 } } }
I20260620 19:14:31.266259 32738 sys_catalog.cc:458] T 00000000000000000000000000000000 P e12cf6e320e74eddba8f8f7c2f7148d4 [sys.catalog]: This master's current role is: LEADER
I20260620 19:14:31.266917 32740 catalog_manager.cc:1347] Generated new cluster ID: 8242c42d9a164acc92e0fb30ada6974f
I20260620 19:14:31.266984 32740 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:14:31.267280 32707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2410319f235046208a95e1e84405ef27 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e12cf6e320e74eddba8f8f7c2f7148d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e12cf6e320e74eddba8f8f7c2f7148d4" member_type: VOTER last_known_addr { host: "127.9.155.126" port: 34667 } } peers { permanent_uuid: "2410319f235046208a95e1e84405ef27" member_type: VOTER last_known_addr { host: "127.9.155.125" port: 45031 } } peers { permanent_uuid: "6e33e378cf6d4b4ebd9eb161f531d50c" member_type: VOTER last_known_addr { host: "127.9.155.124" port: 46467 } } }
I20260620 19:14:31.267401 32707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2410319f235046208a95e1e84405ef27 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:14:31.269001 32747 mvcc.cc:205] Tried to move back new op lower bound from 7299001840581218304 to 7299001840455475200. Current Snapshot: MvccSnapshot[applied={T|T < 7299001840581218304}]
I20260620 19:14:31.278381 32740 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:14:31.282301 32740 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:14:31.304335 32740 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e12cf6e320e74eddba8f8f7c2f7148d4: Generated new TSK 0
I20260620 19:14:31.307423 32740 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 19:14:31.341845 32616 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:14:31.339001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48130 (local address 127.9.155.125:45031)
0620 19:14:31.339136 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:31.339141 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:31.340510 (+ 1369us) server_negotiation.cc:407] Connection header received
0620 19:14:31.340932 (+ 422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:31.340936 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:31.340992 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:31.341043 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:31.341710 (+ 667us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48130: BlockingRecv error: recv got EOF from 127.0.0.1:48130 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260620 19:14:31.343063 32692 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:14:31.339790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38356 (local address 127.9.155.124:46467)
0620 19:14:31.339911 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:31.339915 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:31.340825 (+ 910us) server_negotiation.cc:407] Connection header received
0620 19:14:31.342245 (+ 1420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:31.342249 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:31.342303 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:31.342354 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:31.342931 (+ 577us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38356: BlockingRecv error: recv got EOF from 127.0.0.1:38356 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260620 19:14:31.345397 32540 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:14:31.344011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39928 (local address 127.9.155.126:34667)
0620 19:14:31.344134 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:31.344139 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:31.344503 (+ 364us) server_negotiation.cc:407] Connection header received
0620 19:14:31.344668 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:31.344672 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:31.344724 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:31.344773 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:31.345279 (+ 506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39928: BlockingRecv error: recv got EOF from 127.0.0.1:39928 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260620 19:14:31.350594 32616 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:14:31.348708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48134 (local address 127.9.155.125:45031)
0620 19:14:31.348846 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:31.348851 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:31.349470 (+ 619us) server_negotiation.cc:407] Connection header received
0620 19:14:31.349707 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:31.349711 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:31.349809 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:31.349870 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:31.350478 (+ 608us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48134: BlockingRecv error: recv got EOF from 127.0.0.1:48134 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260620 19:14:31.355967 32616 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:14:31.354262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48148 (local address 127.9.155.125:45031)
0620 19:14:31.354384 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:31.354389 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:31.354992 (+ 603us) server_negotiation.cc:407] Connection header received
0620 19:14:31.355182 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:31.355185 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:31.355241 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:31.355294 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:31.355845 (+ 551us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48148: BlockingRecv error: recv got EOF from 127.0.0.1:48148 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260620 19:14:31.361965 32616 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:14:31.360193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48150 (local address 127.9.155.125:45031)
0620 19:14:31.360407 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:31.360411 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:31.360966 (+ 555us) server_negotiation.cc:407] Connection header received
0620 19:14:31.361170 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:31.361174 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:31.361228 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:31.361278 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:31.361848 (+ 570us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48150: BlockingRecv error: recv got EOF from 127.0.0.1:48150 (error 108)
Metrics: {"server-negotiator.queue_time_us":146}
W20260620 19:14:31.369976 32616 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:14:31.368156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48158 (local address 127.9.155.125:45031)
0620 19:14:31.368340 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:31.368344 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:31.368957 (+ 613us) server_negotiation.cc:407] Connection header received
0620 19:14:31.369153 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:31.369156 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:31.369211 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:31.369267 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:31.369855 (+ 588us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48158: BlockingRecv error: recv got EOF from 127.0.0.1:48158 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
W20260620 19:14:31.382153 32616 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:14:31.380785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48164 (local address 127.9.155.125:45031)
0620 19:14:31.380915 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:31.380919 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:31.381162 (+ 243us) server_negotiation.cc:407] Connection header received
0620 19:14:31.381286 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:31.381289 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:31.381351 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:31.381466 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:31.382034 (+ 568us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48164: BlockingRecv error: recv got EOF from 127.0.0.1:48164 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260620 19:14:31.401564 32616 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:14:31.400324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48168 (local address 127.9.155.125:45031)
0620 19:14:31.400447 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:31.400452 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:31.400735 (+ 283us) server_negotiation.cc:407] Connection header received
0620 19:14:31.400866 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:31.400869 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:31.400930 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:31.401014 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:31.401442 (+ 428us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48168: BlockingRecv error: recv got EOF from 127.0.0.1:48168 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260620 19:14:31.436761 32616 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:14:31.435615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48180 (local address 127.9.155.125:45031)
0620 19:14:31.435772 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:31.435777 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:31.435841 (+ 64us) server_negotiation.cc:407] Connection header received
0620 19:14:31.436032 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:31.436035 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:31.436087 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:31.436157 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:31.436640 (+ 483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48180: BlockingRecv error: recv got EOF from 127.0.0.1:48180 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260620 19:14:31.504148 32616 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:14:31.502876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48186 (local address 127.9.155.125:45031)
0620 19:14:31.503057 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:31.503061 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:31.503127 (+ 66us) server_negotiation.cc:407] Connection header received
0620 19:14:31.503342 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:31.503345 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:31.503400 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:31.503490 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:31.504045 (+ 555us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48186: BlockingRecv error: recv got EOF from 127.0.0.1:48186 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
W20260620 19:14:31.635497 32616 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:14:31.634343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48194 (local address 127.9.155.125:45031)
0620 19:14:31.634503 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:31.634507 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:31.634611 (+ 104us) server_negotiation.cc:407] Connection header received
0620 19:14:31.634759 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:31.634762 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:31.634819 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:31.634894 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:31.635379 (+ 485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48194: BlockingRecv error: recv got EOF from 127.0.0.1:48194 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260620 19:14:31.897593 32616 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:14:31.896326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48204 (local address 127.9.155.125:45031)
0620 19:14:31.896478 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:31.896482 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:31.896749 (+ 267us) server_negotiation.cc:407] Connection header received
0620 19:14:31.896834 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:31.896837 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:31.896893 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:31.896969 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:31.897474 (+ 505us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48204: BlockingRecv error: recv got EOF from 127.0.0.1:48204 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 19:14:32.177030 32704 catalog_manager.cc:1259] Loaded cluster ID: 8242c42d9a164acc92e0fb30ada6974f
I20260620 19:14:32.177119 32704 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6e33e378cf6d4b4ebd9eb161f531d50c: loading cluster ID for follower catalog manager: success
I20260620 19:14:32.178788 32704 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6e33e378cf6d4b4ebd9eb161f531d50c: acquiring CA information for follower catalog manager: success
I20260620 19:14:32.180131 32704 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6e33e378cf6d4b4ebd9eb161f531d50c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 19:14:32.192400 32718 catalog_manager.cc:1259] Loaded cluster ID: 8242c42d9a164acc92e0fb30ada6974f
I20260620 19:14:32.192536 32718 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2410319f235046208a95e1e84405ef27: loading cluster ID for follower catalog manager: success
I20260620 19:14:32.194501 32718 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2410319f235046208a95e1e84405ef27: acquiring CA information for follower catalog manager: success
I20260620 19:14:32.195777 32718 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2410319f235046208a95e1e84405ef27: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 19:14:32.415110 348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:32.411500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48218 (local address 127.9.155.125:45031)
0620 19:14:32.411725 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:32.411728 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:32.411741 (+ 13us) server_negotiation.cc:407] Connection header received
0620 19:14:32.411833 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:32.411836 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:32.411961 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:32.412062 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:14:32.412278 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:32.412832 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:32.413700 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:32.413952 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:32.414044 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:32.414278 (+ 234us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:14:32.414700 (+ 422us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:14:32.414789 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:32.414881 (+ 92us) server_negotiation.cc:299] Negotiation successful
0620 19:14:32.414950 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":84,"threads_started":1}
I20260620 19:14:32.421820 354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:32.418124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38360 (local address 127.9.155.124:46467)
0620 19:14:32.418412 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:32.418415 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:32.418430 (+ 15us) server_negotiation.cc:407] Connection header received
0620 19:14:32.418506 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:32.418508 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:32.418629 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:32.418731 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:14:32.418964 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:32.419482 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:32.420365 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:32.420651 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:32.420733 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:32.421065 (+ 332us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:14:32.421500 (+ 435us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:14:32.421591 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:32.421647 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 19:14:32.421696 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":78,"threads_started":1}
I20260620 19:14:32.428768 354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:32.425223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38372 (local address 127.9.155.124:46467)
0620 19:14:32.425330 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:32.425334 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:32.425565 (+ 231us) server_negotiation.cc:407] Connection header received
0620 19:14:32.425722 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:32.425725 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:32.425772 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:32.425847 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:14:32.426110 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:32.426829 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:32.427660 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:32.427914 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:32.428001 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:32.428324 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:14:32.428412 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:14:32.428472 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:32.428562 (+ 90us) server_negotiation.cc:299] Negotiation successful
0620 19:14:32.428615 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 19:14:32.428901 348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:32.424849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48234 (local address 127.9.155.125:45031)
0620 19:14:32.425058 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:32.425063 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:32.425292 (+ 229us) server_negotiation.cc:407] Connection header received
0620 19:14:32.425464 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:32.425467 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:32.425519 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:32.425580 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:14:32.425899 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:32.426640 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:32.427700 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:32.428039 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:32.428123 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:32.428427 (+ 304us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:14:32.428503 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:14:32.428566 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:32.428702 (+ 136us) server_negotiation.cc:299] Negotiation successful
0620 19:14:32.428758 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260620 19:14:32.429451 361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:32.424825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39938 (local address 127.9.155.126:34667)
0620 19:14:32.425068 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:32.425072 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:32.425087 (+ 15us) server_negotiation.cc:407] Connection header received
0620 19:14:32.425286 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:32.425290 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:32.425455 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:32.425575 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:14:32.425801 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:32.426428 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:32.427549 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:32.427888 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:32.427983 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:32.428316 (+ 333us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:14:32.428671 (+ 355us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:14:32.428741 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:32.429130 (+ 389us) server_negotiation.cc:299] Negotiation successful
0620 19:14:32.429299 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":64,"threads_started":1}
I20260620 19:14:32.432276 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 32468
I20260620 19:14:32.437620 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 32543
I20260620 19:14:32.442663 9837 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task74hYKn/build/debug/bin/kudu with pid 32617
2026-06-20T19:14:32Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2995 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5965 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (277203 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST